[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:34:22 dist-test-slave-fz47 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:34:22 dist-test-slave-fz47 krb5kdc[399](info): set up 2 sockets
Sep 28 02:34:22 dist-test-slave-fz47 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 02:34:23.749061 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:34:23 dist-test-slave-fz47 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:34:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:34:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 02:34:23.851245 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33913
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33913
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:23.926123 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:23.926307 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:23.926327 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:23.927386 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:34:23.927418 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:23.927431 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:23.927441 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:34:23.927460 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:34:23.928800 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:23.928982 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:23.929170 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:23.934091 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:23.934104 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:23.934305 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:23.934134 412 server_base.cc:1034] running on GCE node
I20260928 02:34:23.934535 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:23.934758 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:23.935883 412 hybrid_clock.cc:648] HybridClock initialized: now 1790562863935867 us; error 30 us; skew 500 ppm
Sep 28 02:34:23 dist-test-slave-fz47 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:23.938120 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:34:23.938534 412 webserver.cc:460] Webserver started at http://127.0.98.62:42829/ using document root <none> and password file <none>
I20260928 02:34:23.938747 412 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:23.938789 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:23.938917 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:23.939870 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "68bd48dbbf4c4bc8b0342908f6dcd466"
format_stamp: "Formatted at 2026-09-28 02:34:23 on dist-test-slave-fz47"
server_key: "3654a7befce009343071ed5893e10dff"
server_key_iv: "cf76ba9275dcc02a76ffcc5105562c2b"
server_key_version: "encryptionkey@0"
I20260928 02:34:23.940172 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "68bd48dbbf4c4bc8b0342908f6dcd466"
format_stamp: "Formatted at 2026-09-28 02:34:23 on dist-test-slave-fz47"
server_key: "3654a7befce009343071ed5893e10dff"
server_key_iv: "cf76ba9275dcc02a76ffcc5105562c2b"
server_key_version: "encryptionkey@0"
I20260928 02:34:23.941378 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:34:23.942339 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:23.942634 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:34:23.942739 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "68bd48dbbf4c4bc8b0342908f6dcd466"
format_stamp: "Formatted at 2026-09-28 02:34:23 on dist-test-slave-fz47"
server_key: "3654a7befce009343071ed5893e10dff"
server_key_iv: "cf76ba9275dcc02a76ffcc5105562c2b"
server_key_version: "encryptionkey@0"
I20260928 02:34:23.942816 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:23.953115 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:23.955093 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:23.955219 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:23.958834 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33913
I20260928 02:34:23.958875 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33913 every 8 connection(s)
I20260928 02:34:23.959158 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:34:23.959928 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:23.962064 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68bd48dbbf4c4bc8b0342908f6dcd466: Bootstrap starting.
I20260928 02:34:23.962575 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68bd48dbbf4c4bc8b0342908f6dcd466: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:23.962810 480 log.cc:824] T 00000000000000000000000000000000 P 68bd48dbbf4c4bc8b0342908f6dcd466: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:23.963407 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68bd48dbbf4c4bc8b0342908f6dcd466: No bootstrap required, opened a new log
I20260928 02:34:23.964818 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 412
I20260928 02:34:23.964893 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:34:23.964874 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68bd48dbbf4c4bc8b0342908f6dcd466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bd48dbbf4c4bc8b0342908f6dcd466" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33913 } }
I20260928 02:34:23.965019 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68bd48dbbf4c4bc8b0342908f6dcd466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:23.965077 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68bd48dbbf4c4bc8b0342908f6dcd466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68bd48dbbf4c4bc8b0342908f6dcd466, State: Initialized, Role: FOLLOWER
I20260928 02:34:23.965201 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68bd48dbbf4c4bc8b0342908f6dcd466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bd48dbbf4c4bc8b0342908f6dcd466" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33913 } }
I20260928 02:34:23.965284 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68bd48dbbf4c4bc8b0342908f6dcd466 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:34:23.965322 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68bd48dbbf4c4bc8b0342908f6dcd466 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:34:23.965368 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68bd48dbbf4c4bc8b0342908f6dcd466 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:23.965636 392 external_mini_cluster.cc:1428] Setting key 1c7e8d94d6ca231e1a5bc772b9cb27d5
I20260928 02:34:23.965926 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68bd48dbbf4c4bc8b0342908f6dcd466 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bd48dbbf4c4bc8b0342908f6dcd466" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33913 } }
I20260928 02:34:23.965996 480 leader_election.cc:302] T 00000000000000000000000000000000 P 68bd48dbbf4c4bc8b0342908f6dcd466 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68bd48dbbf4c4bc8b0342908f6dcd466; no voters:
I20260928 02:34:23.966116 480 leader_election.cc:288] T 00000000000000000000000000000000 P 68bd48dbbf4c4bc8b0342908f6dcd466 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:34:23.966157 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68bd48dbbf4c4bc8b0342908f6dcd466 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:23.966269 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68bd48dbbf4c4bc8b0342908f6dcd466 [term 1 LEADER]: Becoming Leader. State: Replica: 68bd48dbbf4c4bc8b0342908f6dcd466, State: Running, Role: LEADER
I20260928 02:34:23.966339 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68bd48dbbf4c4bc8b0342908f6dcd466 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:34:23.966342 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68bd48dbbf4c4bc8b0342908f6dcd466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bd48dbbf4c4bc8b0342908f6dcd466" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33913 } }
I20260928 02:34:23.966687 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68bd48dbbf4c4bc8b0342908f6dcd466 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68bd48dbbf4c4bc8b0342908f6dcd466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bd48dbbf4c4bc8b0342908f6dcd466" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33913 } } }
I20260928 02:34:23.966756 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68bd48dbbf4c4bc8b0342908f6dcd466 [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:23.966781 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68bd48dbbf4c4bc8b0342908f6dcd466 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68bd48dbbf4c4bc8b0342908f6dcd466. Latest consensus state: current_term: 1 leader_uuid: "68bd48dbbf4c4bc8b0342908f6dcd466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68bd48dbbf4c4bc8b0342908f6dcd466" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33913 } } }
I20260928 02:34:23.966866 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68bd48dbbf4c4bc8b0342908f6dcd466 [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:23.967181 489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:34:23.967746 489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:34:23.969422 489 catalog_manager.cc:1347] Generated new cluster ID: d40cc4ff05644a59b77bc359dcdc2a2e
I20260928 02:34:23.969466 489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:34:23.974996 489 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:34:23.975438 489 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 02:34:23 dist-test-slave-fz47 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:23.980597 489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68bd48dbbf4c4bc8b0342908f6dcd466: Generated new TSK 0
I20260928 02:34:23.980711 489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:34:23.982868 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:23.966518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36394 (local address 127.0.98.62:33913)
0928 02:34:23.967191 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:23.967200 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:23.967238 (+ 38us) server_negotiation.cc:407] Connection header received
0928 02:34:23.969554 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:23.969562 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:23.969614 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:23.969740 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:23.971506 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:23.972970 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:23.974362 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:23.975005 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:23.975115 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:23.979140 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:23.979144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:23.979145 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:23.979156 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:23.980946 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:23.981363 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:23.981366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:23.981366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:23.981469 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:23.981848 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:23.981849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:23.981849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:23.982142 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:23.982277 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:23.982506 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 02:34:23.982647 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 02:34:24.008091 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33913
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:24.083976 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:24.084177 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:24.084197 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:24.085330 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:24.085369 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:24.085397 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:34:24.086809 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33913
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:24.087023 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:24.087235 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:24.091871 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:24.091952 506 server_base.cc:1034] running on GCE node
W20260928 02:34:24.091908 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:24.091889 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:24.092255 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:24.092481 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:24.093616 506 hybrid_clock.cc:648] HybridClock initialized: now 1790562864093602 us; error 29 us; skew 500 ppm
Sep 28 02:34:24 dist-test-slave-fz47 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:24.095945 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 02:34:24.096359 506 webserver.cc:460] Webserver started at http://127.0.98.1:43925/ using document root <none> and password file <none>
I20260928 02:34:24.096561 506 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:24.096602 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:24.096716 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:24.097756 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-0/data/instance:
uuid: "33ed604b9b3c452f8974913ea2f1178e"
format_stamp: "Formatted at 2026-09-28 02:34:24 on dist-test-slave-fz47"
server_key: "61cba23b28ddd43fde0e869c2d543efd"
server_key_iv: "f91dde4209f6982b77a0318518fd1aee"
server_key_version: "encryptionkey@0"
I20260928 02:34:24.098078 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-0/wal/instance:
uuid: "33ed604b9b3c452f8974913ea2f1178e"
format_stamp: "Formatted at 2026-09-28 02:34:24 on dist-test-slave-fz47"
server_key: "61cba23b28ddd43fde0e869c2d543efd"
server_key_iv: "f91dde4209f6982b77a0318518fd1aee"
server_key_version: "encryptionkey@0"
I20260928 02:34:24.099273 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:34:24.099974 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:24.100144 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:24.100224 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "33ed604b9b3c452f8974913ea2f1178e"
format_stamp: "Formatted at 2026-09-28 02:34:24 on dist-test-slave-fz47"
server_key: "61cba23b28ddd43fde0e869c2d543efd"
server_key_iv: "f91dde4209f6982b77a0318518fd1aee"
server_key_version: "encryptionkey@0"
I20260928 02:34:24.100274 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:24.116947 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:24.118937 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:24.119071 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:24.119254 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:34:24.119503 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:24.119536 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:24.119556 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:24.119570 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:24.124738 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35947
I20260928 02:34:24.124778 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35947 every 8 connection(s)
I20260928 02:34:24.125077 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 02:34:24 dist-test-slave-fz47 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:24.131982 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 506
I20260928 02:34:24.132066 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-0/wal/instance
I20260928 02:34:24.132195 392 external_mini_cluster.cc:1428] Setting key 4be1881102f7fe15f424acb6077e14d7
I20260928 02:34:24.133463 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:24.125727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59225 (local address 127.0.98.62:33913)
0928 02:34:24.125788 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:24.125790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:24.125969 (+ 179us) server_negotiation.cc:407] Connection header received
0928 02:34:24.126465 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:24.126466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:24.126478 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:24.126543 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:24.127430 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:24.127927 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:24.128878 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:24.129077 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:24.129130 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:24.131040 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:24.131042 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:24.131042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:24.131049 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:24.132158 (+ 1109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:24.132552 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:24.132553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:24.132554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:24.132611 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:24.132819 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:24.132820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:24.132820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:24.132969 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:24.133045 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:24.133280 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 02:34:24.133377 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 02:34:24.133531 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:24.125696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33913 (local address 127.0.98.1:59225)
0928 02:34:24.125857 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:24.125891 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:24.126373 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:24.126613 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:24.126616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:24.127155 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:24.127337 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:24.127341 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:24.128010 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:24.128011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:24.128796 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:24.128798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:24.129141 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:24.129142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:24.129279 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:24.129685 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:24.129693 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:24.130946 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:24.132236 (+ 1290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:24.132243 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:24.132245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:24.132496 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:24.132661 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:24.132662 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:24.132662 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:24.132767 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:24.133033 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:24.133037 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:24.133145 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 02:34:24.133223 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 02:34:24.133378 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260928 02:34:24.133917 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33913
I20260928 02:34:24.134014 635 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:24.134194 635 heartbeater.cc:511] Master 127.0.98.62:33913 requested a full tablet report, sending...
I20260928 02:34:24.134842 444 ts_manager.cc:194] Registered new tserver with Master: 33ed604b9b3c452f8974913ea2f1178e (127.0.98.1:35947)
I20260928 02:34:24.135430 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59225
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 02:34:24.152827 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33913
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:24.229003 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:24.229187 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:24.229208 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:24.230345 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:24.230389 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:24.230414 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:34:24.231796 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33913
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:24.231995 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:24.232245 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:24.236958 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:24.237040 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:24.237093 640 server_base.cc:1034] running on GCE node
W20260928 02:34:24.236958 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:24.237344 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:24.237541 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:24.238672 640 hybrid_clock.cc:648] HybridClock initialized: now 1790562864238660 us; error 28 us; skew 500 ppm
Sep 28 02:34:24 dist-test-slave-fz47 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:24.240942 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 02:34:24.241345 640 webserver.cc:460] Webserver started at http://127.0.98.2:45079/ using document root <none> and password file <none>
I20260928 02:34:24.241518 640 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:24.241555 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:24.241642 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:24.242605 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-1/data/instance:
uuid: "cb95ce64fbae483aa322f8ac35c0b322"
format_stamp: "Formatted at 2026-09-28 02:34:24 on dist-test-slave-fz47"
server_key: "fc3783f34a56633c7113af453c647069"
server_key_iv: "1bb4103021581eb6998e782d8e1458d5"
server_key_version: "encryptionkey@0"
I20260928 02:34:24.242892 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cb95ce64fbae483aa322f8ac35c0b322"
format_stamp: "Formatted at 2026-09-28 02:34:24 on dist-test-slave-fz47"
server_key: "fc3783f34a56633c7113af453c647069"
server_key_iv: "1bb4103021581eb6998e782d8e1458d5"
server_key_version: "encryptionkey@0"
I20260928 02:34:24.243999 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:34:24.244611 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:24.244788 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:24.244875 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "cb95ce64fbae483aa322f8ac35c0b322"
format_stamp: "Formatted at 2026-09-28 02:34:24 on dist-test-slave-fz47"
server_key: "fc3783f34a56633c7113af453c647069"
server_key_iv: "1bb4103021581eb6998e782d8e1458d5"
server_key_version: "encryptionkey@0"
I20260928 02:34:24.244953 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:24.259322 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:24.261391 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:24.261497 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:24.261672 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:34:24.261915 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:24.261945 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:24.261963 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:24.261979 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:24.267416 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33139
I20260928 02:34:24.267477 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33139 every 8 connection(s)
I20260928 02:34:24.267755 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 02:34:24 dist-test-slave-fz47 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:24.275890 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:24.268350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51685 (local address 127.0.98.62:33913)
0928 02:34:24.268416 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:24.268418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:24.268589 (+ 171us) server_negotiation.cc:407] Connection header received
0928 02:34:24.269085 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:24.269086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:24.269095 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:24.269143 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:24.270050 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:24.270535 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:24.271421 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:24.271591 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:24.271633 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:24.273898 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:24.273900 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:24.273900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:24.273907 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:24.274938 (+ 1031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:24.275254 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:24.275255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:24.275255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:24.275296 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:24.275453 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:24.275454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:24.275454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:24.275541 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:24.275583 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:24.275773 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 02:34:24.275829 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:34:24.276077 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:24.268329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33913 (local address 127.0.98.2:51685)
0928 02:34:24.268494 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:24.268528 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:24.269005 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:24.269197 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:24.269200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:24.269769 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:24.269950 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:24.269955 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:24.270601 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:24.270602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:24.271363 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:24.271365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:24.271639 (+ 274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:24.271639 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:24.271727 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:24.272503 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:24.272511 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:24.273818 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:24.275003 (+ 1185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:24.275009 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:24.275010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:24.275201 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:24.275337 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:24.275337 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:24.275338 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:24.275416 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:24.275580 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:24.275582 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:24.275685 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 02:34:24.275758 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:34:24.275865 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260928 02:34:24.276404 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 640
I20260928 02:34:24.276489 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-1/wal/instance
I20260928 02:34:24.276634 392 external_mini_cluster.cc:1428] Setting key d61da9d9607c49165b39856f164e5a43
I20260928 02:34:24.276921 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33913
I20260928 02:34:24.277015 769 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:24.277184 769 heartbeater.cc:511] Master 127.0.98.62:33913 requested a full tablet report, sending...
I20260928 02:34:24.277570 444 ts_manager.cc:194] Registered new tserver with Master: cb95ce64fbae483aa322f8ac35c0b322 (127.0.98.2:33139)
I20260928 02:34:24.277910 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51685
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 02:34:24.297312 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33913
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:24.375120 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:24.375329 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:24.375352 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:24.376475 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:24.376513 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:24.376537 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:34:24.377885 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33913
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:24.378046 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:24.378206 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:24.382661 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:24.382663 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:24.382733 774 server_base.cc:1034] running on GCE node
W20260928 02:34:24.382700 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:24.383055 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:24.383257 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:24.384403 774 hybrid_clock.cc:648] HybridClock initialized: now 1790562864384392 us; error 31 us; skew 500 ppm
Sep 28 02:34:24 dist-test-slave-fz47 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:24.386720 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 02:34:24.387113 774 webserver.cc:460] Webserver started at http://127.0.98.3:43257/ using document root <none> and password file <none>
I20260928 02:34:24.387290 774 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:24.387333 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:24.387445 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:24.388360 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-2/data/instance:
uuid: "e29743b275f24b6f8953e1e4736f0ca2"
format_stamp: "Formatted at 2026-09-28 02:34:24 on dist-test-slave-fz47"
server_key: "96a59ad25e9853fff6e5d02d74629b98"
server_key_iv: "d2efec812dacf216fb102670bf3bea09"
server_key_version: "encryptionkey@0"
I20260928 02:34:24.388674 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e29743b275f24b6f8953e1e4736f0ca2"
format_stamp: "Formatted at 2026-09-28 02:34:24 on dist-test-slave-fz47"
server_key: "96a59ad25e9853fff6e5d02d74629b98"
server_key_iv: "d2efec812dacf216fb102670bf3bea09"
server_key_version: "encryptionkey@0"
I20260928 02:34:24.389848 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:34:24.390447 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:24.390580 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:24.390661 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "e29743b275f24b6f8953e1e4736f0ca2"
format_stamp: "Formatted at 2026-09-28 02:34:24 on dist-test-slave-fz47"
server_key: "96a59ad25e9853fff6e5d02d74629b98"
server_key_iv: "d2efec812dacf216fb102670bf3bea09"
server_key_version: "encryptionkey@0"
I20260928 02:34:24.390714 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:24.414528 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:24.416492 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:24.416618 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:24.416807 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:34:24.417078 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:24.417109 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:24.417135 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:24.417153 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:24.422922 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35301
I20260928 02:34:24.422976 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35301 every 8 connection(s)
I20260928 02:34:24.423238 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 02:34:24 dist-test-slave-fz47 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:24.430759 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:24.423854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50697 (local address 127.0.98.62:33913)
0928 02:34:24.423904 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:24.423906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:24.424083 (+ 177us) server_negotiation.cc:407] Connection header received
0928 02:34:24.424506 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:24.424506 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:24.424515 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:24.424567 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:24.425383 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:24.425862 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:24.426769 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:24.426952 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:24.427012 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:24.428783 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:24.428784 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:24.428785 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:24.428791 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:24.429812 (+ 1021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:24.430130 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:24.430131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:24.430131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:24.430171 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:24.430339 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:24.430339 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:24.430340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:24.430422 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:24.430463 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:24.430650 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 02:34:24.430700 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 02:34:24.430904 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:24.423833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33913 (local address 127.0.98.3:50697)
0928 02:34:24.423971 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:24.424017 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:24.424436 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:24.424619 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:24.424622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:24.425124 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:24.425318 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:24.425322 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:24.425922 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:24.425923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:24.426706 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:24.426708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:24.427025 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:24.427026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:24.427114 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:24.427523 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:24.427530 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:24.428711 (+ 1181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:24.429869 (+ 1158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:24.429873 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:24.429875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:24.430078 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:24.430213 (+ 135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:24.430214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:24.430214 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:24.430299 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:24.430461 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:24.430464 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:24.430562 (+ 98us) client_negotiation.cc:769] Sending connection context
0928 02:34:24.430633 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 02:34:24.430736 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":65,"threads_started":1}
I20260928 02:34:24.431217 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 774
I20260928 02:34:24.431324 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790562862804661-392-0/minicluster-data/ts-2/wal/instance
I20260928 02:34:24.431357 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33913
I20260928 02:34:24.431447 903 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:24.431493 392 external_mini_cluster.cc:1428] Setting key bc8fb0f874b279d5dccffa075e48b1b2
I20260928 02:34:24.431623 903 heartbeater.cc:511] Master 127.0.98.62:33913 requested a full tablet report, sending...
I20260928 02:34:24.431959 444 ts_manager.cc:194] Registered new tserver with Master: e29743b275f24b6f8953e1e4736f0ca2 (127.0.98.3:35301)
I20260928 02:34:24.432379 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50697
I20260928 02:34:24.432471 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:34:24 dist-test-slave-fz47 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562864, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:34:24 dist-test-slave-fz47 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562864, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 02:34:24.452787 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:24.445946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38738 (local address 127.0.98.1:35947)
0928 02:34:24.446093 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:24.446097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:24.446107 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:34:24.446128 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:24.446130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:24.446141 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:24.446202 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:24.446694 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:24.447438 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:24.448093 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:24.448370 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:24.448415 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:24.450669 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:24.450675 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:24.450675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:24.450682 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:24.452004 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:24.452237 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:24.452239 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:24.452239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:24.452285 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:24.452397 (+ 112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:24.452398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:24.452398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:24.452499 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:24.452587 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:24.452637 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 02:34:24.452693 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
W20260928 02:34:24.453120 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38738
Sep 28 02:34:24 dist-test-slave-fz47 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:34:24 dist-test-slave-fz47 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 02:34:24.471828 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:24.465641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38744 (local address 127.0.98.1:35947)
0928 02:34:24.465689 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:24.465690 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:24.465748 (+ 58us) server_negotiation.cc:407] Connection header received
0928 02:34:24.465818 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:24.465819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:24.465835 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:24.465881 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:24.466439 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:24.466981 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:24.467746 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:24.467983 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:24.468034 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:24.469926 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:24.469927 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:24.469927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:24.469933 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:24.471111 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:24.471351 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:24.471351 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:24.471351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:24.471395 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:24.471514 (+ 119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:24.471515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:24.471515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:24.471585 (+ 70us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:24.471663 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:24.471706 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 02:34:24.471768 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 02:34:24.472252 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 506
I20260928 02:34:24.476214 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 640
I20260928 02:34:24.481000 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 774
I20260928 02:34:24.484941 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 412
2026-09-28T02:34:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1685 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:34:24 dist-test-slave-fz47 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:34:24 dist-test-slave-fz47 krb5kdc[917](info): set up 2 sockets
Sep 28 02:34:24 dist-test-slave-fz47 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260928 02:34:25.432557 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.003s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:34:25 dist-test-slave-fz47 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:34:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:34:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 02:34:25.535629 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34723
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34723
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:25.612188 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:25.612380 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:25.612407 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:25.613567 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:34:25.613605 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:25.613621 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:25.613637 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:34:25.613653 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:34:25.615046 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:25.615270 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:25.615460 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:25.620059 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:25.620091 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:25.620144 930 server_base.cc:1034] running on GCE node
W20260928 02:34:25.620086 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:25.620433 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:25.620684 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:25.621815 930 hybrid_clock.cc:648] HybridClock initialized: now 1790562865621794 us; error 39 us; skew 500 ppm
Sep 28 02:34:25 dist-test-slave-fz47 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:25.627014 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:34:25.627424 930 webserver.cc:460] Webserver started at http://127.0.98.62:42865/ using document root <none> and password file <none>
I20260928 02:34:25.627610 930 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:25.627650 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:25.627769 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:25.628767 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "a0c356241576442a935c1cc6d4cda3a1"
format_stamp: "Formatted at 2026-09-28 02:34:25 on dist-test-slave-fz47"
server_key: "74b0c34f9990c3d298b790b5705df9aa"
server_key_iv: "47db7dfdad153c25f06117ac08a16fab"
server_key_version: "encryptionkey@0"
I20260928 02:34:25.629081 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "a0c356241576442a935c1cc6d4cda3a1"
format_stamp: "Formatted at 2026-09-28 02:34:25 on dist-test-slave-fz47"
server_key: "74b0c34f9990c3d298b790b5705df9aa"
server_key_iv: "47db7dfdad153c25f06117ac08a16fab"
server_key_version: "encryptionkey@0"
I20260928 02:34:25.630281 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:34:25.630863 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:25.631031 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:25.631111 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "a0c356241576442a935c1cc6d4cda3a1"
format_stamp: "Formatted at 2026-09-28 02:34:25 on dist-test-slave-fz47"
server_key: "74b0c34f9990c3d298b790b5705df9aa"
server_key_iv: "47db7dfdad153c25f06117ac08a16fab"
server_key_version: "encryptionkey@0"
I20260928 02:34:25.631163 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:25.661897 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:25.663622 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:25.663731 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:25.667212 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34723
I20260928 02:34:25.667258 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34723 every 8 connection(s)
I20260928 02:34:25.667560 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:34:25.668320 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:25.669379 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 930
I20260928 02:34:25.669453 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:34:25.669581 392 external_mini_cluster.cc:1428] Setting key 5e9ae965b3bae9f8b29dba9f5a77d380
I20260928 02:34:25.670996 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0c356241576442a935c1cc6d4cda3a1: Bootstrap starting.
I20260928 02:34:25.671612 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0c356241576442a935c1cc6d4cda3a1: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:25.671895 998 log.cc:824] T 00000000000000000000000000000000 P a0c356241576442a935c1cc6d4cda3a1: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:25.672575 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0c356241576442a935c1cc6d4cda3a1: No bootstrap required, opened a new log
I20260928 02:34:25.674646 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0c356241576442a935c1cc6d4cda3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0c356241576442a935c1cc6d4cda3a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34723 } }
I20260928 02:34:25.674772 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0c356241576442a935c1cc6d4cda3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:25.674793 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0c356241576442a935c1cc6d4cda3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0c356241576442a935c1cc6d4cda3a1, State: Initialized, Role: FOLLOWER
I20260928 02:34:25.674901 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0c356241576442a935c1cc6d4cda3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0c356241576442a935c1cc6d4cda3a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34723 } }
I20260928 02:34:25.674970 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0c356241576442a935c1cc6d4cda3a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:34:25.674998 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0c356241576442a935c1cc6d4cda3a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:34:25.675043 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0c356241576442a935c1cc6d4cda3a1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:25.675519 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0c356241576442a935c1cc6d4cda3a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0c356241576442a935c1cc6d4cda3a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34723 } }
I20260928 02:34:25.675609 998 leader_election.cc:302] T 00000000000000000000000000000000 P a0c356241576442a935c1cc6d4cda3a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0c356241576442a935c1cc6d4cda3a1; no voters:
I20260928 02:34:25.675760 998 leader_election.cc:288] T 00000000000000000000000000000000 P a0c356241576442a935c1cc6d4cda3a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:34:25.675796 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0c356241576442a935c1cc6d4cda3a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:25.675904 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0c356241576442a935c1cc6d4cda3a1 [term 1 LEADER]: Becoming Leader. State: Replica: a0c356241576442a935c1cc6d4cda3a1, State: Running, Role: LEADER
I20260928 02:34:25.675997 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0c356241576442a935c1cc6d4cda3a1 [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 02:34:25 dist-test-slave-fz47 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:25.676051 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0c356241576442a935c1cc6d4cda3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0c356241576442a935c1cc6d4cda3a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34723 } }
I20260928 02:34:25.676441 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0c356241576442a935c1cc6d4cda3a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0c356241576442a935c1cc6d4cda3a1. Latest consensus state: current_term: 1 leader_uuid: "a0c356241576442a935c1cc6d4cda3a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0c356241576442a935c1cc6d4cda3a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34723 } } }
I20260928 02:34:25.676453 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0c356241576442a935c1cc6d4cda3a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0c356241576442a935c1cc6d4cda3a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0c356241576442a935c1cc6d4cda3a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34723 } } }
I20260928 02:34:25.676527 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0c356241576442a935c1cc6d4cda3a1 [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:25.676553 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0c356241576442a935c1cc6d4cda3a1 [sys.catalog]: This master's current role is: LEADER
W20260928 02:34:25.677561 1018 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a0c356241576442a935c1cc6d4cda3a1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:34:25.677616 1018 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:34:25.677657 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:34:25.677789 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:34:25.679327 1012 catalog_manager.cc:1347] Generated new cluster ID: e9a7f418437941b2b2f4498819609255
I20260928 02:34:25.679378 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:34:25.680047 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:25.670049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52112 (local address 127.0.98.62:34723)
0928 02:34:25.670224 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:25.670234 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:25.670267 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:34:25.670745 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:25.670754 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:25.670825 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:25.670983 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:25.671501 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:25.672801 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:25.673546 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:25.674170 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:25.674286 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:25.676558 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:25.676562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:25.676562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:25.676573 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:25.678318 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:25.678682 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:25.678684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:25.678685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:25.678788 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:25.678976 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:25.678977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:25.678977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:25.679345 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:25.679551 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:25.679762 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 02:34:25.679897 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":73,"threads_started":1}
I20260928 02:34:25.686115 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:34:25.686761 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:34:25.692294 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0c356241576442a935c1cc6d4cda3a1: Generated new TSK 0
I20260928 02:34:25.692444 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 02:34:25.715641 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34723
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:25.792431 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:25.792620 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:25.792642 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:25.793810 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:25.793848 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:25.793874 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:34:25.795246 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34723
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:25.795459 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:25.795657 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:34:25.800156 1024 server_base.cc:1034] running on GCE node
W20260928 02:34:25.800141 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:25.800254 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:25.800231 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:25.800549 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:25.800755 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:25.801896 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790562865801877 us; error 30 us; skew 500 ppm
Sep 28 02:34:25 dist-test-slave-fz47 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:25.804009 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 02:34:25.804382 1024 webserver.cc:460] Webserver started at http://127.0.98.1:39779/ using document root <none> and password file <none>
I20260928 02:34:25.804577 1024 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:25.804621 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:25.804738 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:25.805739 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-0/data/instance:
uuid: "5a3d698d83be4eee82c4109c37104f41"
format_stamp: "Formatted at 2026-09-28 02:34:25 on dist-test-slave-fz47"
server_key: "c5b0073379a808420abe6f67bf1fbb15"
server_key_iv: "ec25179d8def840ca43173a90697f502"
server_key_version: "encryptionkey@0"
I20260928 02:34:25.806044 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5a3d698d83be4eee82c4109c37104f41"
format_stamp: "Formatted at 2026-09-28 02:34:25 on dist-test-slave-fz47"
server_key: "c5b0073379a808420abe6f67bf1fbb15"
server_key_iv: "ec25179d8def840ca43173a90697f502"
server_key_version: "encryptionkey@0"
I20260928 02:34:25.807452 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:34:25.808030 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:25.808169 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:25.808243 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "5a3d698d83be4eee82c4109c37104f41"
format_stamp: "Formatted at 2026-09-28 02:34:25 on dist-test-slave-fz47"
server_key: "c5b0073379a808420abe6f67bf1fbb15"
server_key_iv: "ec25179d8def840ca43173a90697f502"
server_key_version: "encryptionkey@0"
I20260928 02:34:25.808305 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:25.827046 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:25.828935 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:25.829064 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:25.829290 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:34:25.829588 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:25.829618 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:25.829643 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:25.829660 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:25.834846 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40293
I20260928 02:34:25.834932 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40293 every 8 connection(s)
I20260928 02:34:25.835170 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
I20260928 02:34:25.839510 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 1024
I20260928 02:34:25.839577 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-0/wal/instance
I20260928 02:34:25.839706 392 external_mini_cluster.cc:1428] Setting key ef9a2d19538222682094454d9535913f
Sep 28 02:34:25 dist-test-slave-fz47 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:25.843701 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:25.835756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44919 (local address 127.0.98.62:34723)
0928 02:34:25.835821 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:25.835823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:25.835987 (+ 164us) server_negotiation.cc:407] Connection header received
0928 02:34:25.836425 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:25.836426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:25.836435 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:25.836478 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:25.837328 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:25.837801 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:25.838785 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:25.838983 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:25.839026 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:25.841355 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:25.841356 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:25.841357 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:25.841364 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:25.842551 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:25.842893 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:25.842894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:25.842894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:25.842946 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:25.843149 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:25.843150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:25.843150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:25.843249 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:25.843304 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:25.843539 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 02:34:25.843617 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:34:25.843847 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:25.835741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34723 (local address 127.0.98.1:44919)
0928 02:34:25.835875 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:25.835915 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:25.836354 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:25.836534 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:25.836537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:25.837050 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:25.837231 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:25.837236 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:25.837899 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:25.837904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:25.838707 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:25.838710 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:25.839036 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:25.839037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:25.839134 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:25.839603 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:25.839613 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:25.841246 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:25.842613 (+ 1367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:25.842618 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:25.842619 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:25.842810 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:25.843002 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:25.843003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:25.843003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:25.843094 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:25.843300 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:25.843306 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:25.843428 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 02:34:25.843513 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 02:34:25.843652 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
I20260928 02:34:25.844321 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34723
I20260928 02:34:25.844408 1153 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:25.844564 1153 heartbeater.cc:511] Master 127.0.98.62:34723 requested a full tablet report, sending...
I20260928 02:34:25.845070 962 ts_manager.cc:194] Registered new tserver with Master: 5a3d698d83be4eee82c4109c37104f41 (127.0.98.1:40293)
I20260928 02:34:25.845963 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44919
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 02:34:25.861016 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34723
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:25.938192 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:25.938380 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:25.938401 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:25.939482 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:25.939517 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:25.939540 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:34:25.940864 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34723
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:25.941045 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:25.941210 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:25.945792 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:25.945973 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:25.945866 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:25.945931 1158 server_base.cc:1034] running on GCE node
I20260928 02:34:25.946238 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:25.946450 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:25.947587 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790562865947572 us; error 26 us; skew 500 ppm
Sep 28 02:34:25 dist-test-slave-fz47 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:25.949843 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 02:34:25.950230 1158 webserver.cc:460] Webserver started at http://127.0.98.2:41007/ using document root <none> and password file <none>
I20260928 02:34:25.950434 1158 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:25.950479 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:25.950598 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:25.951567 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-1/data/instance:
uuid: "00ee477367894eb5bc4ab5b40a72be22"
format_stamp: "Formatted at 2026-09-28 02:34:25 on dist-test-slave-fz47"
server_key: "10757845b6e37fda1cc07a054daf3b06"
server_key_iv: "c8eb6766fb9891494baece382d520309"
server_key_version: "encryptionkey@0"
I20260928 02:34:25.951869 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-1/wal/instance:
uuid: "00ee477367894eb5bc4ab5b40a72be22"
format_stamp: "Formatted at 2026-09-28 02:34:25 on dist-test-slave-fz47"
server_key: "10757845b6e37fda1cc07a054daf3b06"
server_key_iv: "c8eb6766fb9891494baece382d520309"
server_key_version: "encryptionkey@0"
I20260928 02:34:25.953015 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:34:25.953668 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:25.953805 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:25.953881 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "00ee477367894eb5bc4ab5b40a72be22"
format_stamp: "Formatted at 2026-09-28 02:34:25 on dist-test-slave-fz47"
server_key: "10757845b6e37fda1cc07a054daf3b06"
server_key_iv: "c8eb6766fb9891494baece382d520309"
server_key_version: "encryptionkey@0"
I20260928 02:34:25.953943 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:25.984400 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:25.986330 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:25.986466 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:25.986678 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:34:25.986967 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:25.986999 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:25.987027 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:25.987046 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:25.992147 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46121
I20260928 02:34:25.992205 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46121 every 8 connection(s)
I20260928 02:34:25.992475 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
I20260928 02:34:25.995244 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 1158
I20260928 02:34:25.995335 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-1/wal/instance
I20260928 02:34:25.995499 392 external_mini_cluster.cc:1428] Setting key 3a5f526f9cc955f036ea502f6785112c
Sep 28 02:34:25 dist-test-slave-fz47 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:26.001646 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:25.993080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35629 (local address 127.0.98.62:34723)
0928 02:34:25.993122 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:25.993124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:25.993334 (+ 210us) server_negotiation.cc:407] Connection header received
0928 02:34:25.993806 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:25.993807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:25.993816 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:25.993867 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:25.994699 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:25.995264 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:25.996279 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:25.996546 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:25.996613 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:25.998832 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:25.998833 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:25.998833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:25.998839 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:26.000281 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:26.000754 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:26.000755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:26.000755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:26.000807 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:26.001058 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:26.001058 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:26.001058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:26.001149 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:26.001192 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:26.001524 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 02:34:26.001585 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:34:26.001896 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:25.993052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34723 (local address 127.0.98.2:35629)
0928 02:34:25.993191 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:25.993232 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:25.993731 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:25.993922 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:25.993925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:25.994443 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:25.994628 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:25.994632 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:25.995355 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:25.995356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:25.996195 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:25.996198 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:25.996642 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:25.996643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:25.996758 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:25.997251 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:25.997280 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:25.998749 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:26.000372 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:26.000379 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:26.000382 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:26.000682 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:26.000859 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:26.000860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:26.000860 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:26.001000 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:26.001200 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:26.001205 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:26.001396 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 02:34:26.001511 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 02:34:26.001667 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
I20260928 02:34:26.002352 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34723
I20260928 02:34:26.002440 1287 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:26.002604 1287 heartbeater.cc:511] Master 127.0.98.62:34723 requested a full tablet report, sending...
I20260928 02:34:26.002945 962 ts_manager.cc:194] Registered new tserver with Master: 00ee477367894eb5bc4ab5b40a72be22 (127.0.98.2:46121)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 02:34:26.003420 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 02:34:26.017614 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34723
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:26.094661 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:26.094858 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:26.094883 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:26.096006 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:26.096042 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:26.096067 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:34:26.097437 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34723
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:26.097616 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:26.097791 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:26.102521 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:26.102528 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:26.102558 1292 server_base.cc:1034] running on GCE node
W20260928 02:34:26.102581 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:26.102936 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:26.103170 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:26.104292 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790562866104281 us; error 26 us; skew 500 ppm
Sep 28 02:34:26 dist-test-slave-fz47 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:26.106631 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 02:34:26.107056 1292 webserver.cc:460] Webserver started at http://127.0.98.3:46071/ using document root <none> and password file <none>
I20260928 02:34:26.107252 1292 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:26.107297 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:26.107416 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:26.108413 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-2/data/instance:
uuid: "fa22b463f48747368debdd4997b74d1b"
format_stamp: "Formatted at 2026-09-28 02:34:26 on dist-test-slave-fz47"
server_key: "a2d91c216b6d44b08c44967778399df6"
server_key_iv: "b4300fa03f9fc70da966abf99e48970c"
server_key_version: "encryptionkey@0"
I20260928 02:34:26.108736 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fa22b463f48747368debdd4997b74d1b"
format_stamp: "Formatted at 2026-09-28 02:34:26 on dist-test-slave-fz47"
server_key: "a2d91c216b6d44b08c44967778399df6"
server_key_iv: "b4300fa03f9fc70da966abf99e48970c"
server_key_version: "encryptionkey@0"
I20260928 02:34:26.109973 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:34:26.110601 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:26.110741 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:34:26.110818 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "fa22b463f48747368debdd4997b74d1b"
format_stamp: "Formatted at 2026-09-28 02:34:26 on dist-test-slave-fz47"
server_key: "a2d91c216b6d44b08c44967778399df6"
server_key_iv: "b4300fa03f9fc70da966abf99e48970c"
server_key_version: "encryptionkey@0"
I20260928 02:34:26.110908 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:26.134045 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:26.136054 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:26.136181 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:26.136389 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:34:26.136653 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:26.136683 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:26.136711 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:26.136735 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:26.142419 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36105
I20260928 02:34:26.142457 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36105 every 8 connection(s)
I20260928 02:34:26.142802 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 02:34:26 dist-test-slave-fz47 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:26.151317 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:26.143458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40721 (local address 127.0.98.62:34723)
0928 02:34:26.143511 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:26.143512 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:26.143693 (+ 181us) server_negotiation.cc:407] Connection header received
0928 02:34:26.144185 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:26.144186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:26.144195 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:26.144261 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:26.145161 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:26.145695 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:26.146649 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:26.146823 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:26.146866 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:26.148930 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:26.148932 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:26.148932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:26.148940 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:26.150137 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:26.150504 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:26.150505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:26.150506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:26.150561 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:26.150816 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:26.150816 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:26.150816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:26.150909 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:26.150953 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:26.151197 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 02:34:26.151255 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 02:34:26.151502 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:26.143432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34723 (local address 127.0.98.3:40721)
0928 02:34:26.143572 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:26.143613 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:26.144109 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:26.144355 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:26.144359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:26.144899 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:26.145091 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:26.145095 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:26.145783 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:26.145784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:26.146588 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:26.146590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:26.146875 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:26.146876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:26.146968 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:26.147490 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:26.147498 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:26.148841 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:26.150202 (+ 1361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:26.150207 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:26.150209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:26.150422 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:26.150642 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:26.150643 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:26.150643 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:26.150765 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:26.150974 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:26.150978 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:26.151097 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 02:34:26.151182 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 02:34:26.151291 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260928 02:34:26.151790 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 1292
I20260928 02:34:26.151898 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790562862804661-392-0/minicluster-data/ts-2/wal/instance
I20260928 02:34:26.151953 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34723
I20260928 02:34:26.152040 1421 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:26.152062 392 external_mini_cluster.cc:1428] Setting key 88f3360b41476e9aa66ebc5d5213b7dc
I20260928 02:34:26.152213 1421 heartbeater.cc:511] Master 127.0.98.62:34723 requested a full tablet report, sending...
I20260928 02:34:26.152588 962 ts_manager.cc:194] Registered new tserver with Master: fa22b463f48747368debdd4997b74d1b (127.0.98.3:36105)
I20260928 02:34:26.152817 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:34:26.153028 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40721
I20260928 02:34:26.159161 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:26.154295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52118 (local address 127.0.98.62:34723)
0928 02:34:26.154343 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:26.154345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:26.154447 (+ 102us) server_negotiation.cc:407] Connection header received
0928 02:34:26.154482 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:26.154482 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:26.154490 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:26.154531 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:26.154982 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:26.155419 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:26.156039 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:26.156246 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:26.156293 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:26.157246 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:26.157247 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:26.157247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:26.157251 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:26.158397 (+ 1146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:26.158665 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:26.158665 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:26.158666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:26.158705 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:26.158814 (+ 109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:26.158815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:26.158815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:26.158923 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:26.158990 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:26.159059 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 02:34:26.159107 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 02:34:26.161080 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:34:26.162583 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:34:26.168260 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:26.164549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36105 (local address 127.0.0.1:60618)
0928 02:34:26.164901 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:26.164911 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:26.164978 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:26.165185 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:26.165188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:26.165190 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:26.165244 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:26.165246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:26.166150 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:26.166151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:26.167136 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:26.167138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:26.168000 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:26.168001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:26.168073 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:26.168085 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 02:34:26.168142 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 02:34:26.168190 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":2,"spinlock_wait_cycles":896,"thread_start_us":54,"threads_started":1}
I20260928 02:34:26.168263 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:26.164697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60618 (local address 127.0.98.3:36105)
0928 02:34:26.164837 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:26.164840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:26.165055 (+ 215us) server_negotiation.cc:407] Connection header received
0928 02:34:26.165090 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:26.165092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:26.165106 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:26.165168 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:26.165328 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:26.166064 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:26.167245 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:26.167937 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:26.167991 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:26.168044 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:26.168159 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 02:34:26.168196 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260928 02:34:26.168610 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:26.164549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40293 (local address 127.0.0.1:59030)
0928 02:34:26.164868 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:26.164904 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:26.165000 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:26.165286 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:26.165286 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:26.165289 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:26.165331 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:26.165333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:26.166649 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:26.166650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:26.167536 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:26.167538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:26.168391 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:26.168392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:26.168451 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:26.168455 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:34:26.168507 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 02:34:26.168542 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":37,"thread_start_us":52,"threads_started":1}
I20260928 02:34:26.168654 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:26.164635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59030 (local address 127.0.98.1:40293)
0928 02:34:26.164770 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:26.164775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:26.164971 (+ 196us) server_negotiation.cc:407] Connection header received
0928 02:34:26.165087 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:26.165090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:26.165108 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:26.165185 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:26.165417 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:26.166534 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:26.167603 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:26.168239 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:26.168306 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:26.168358 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:26.168536 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 02:34:26.168580 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
I20260928 02:34:26.168828 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:26.164788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46121 (local address 127.0.0.1:47310)
0928 02:34:26.165086 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:26.165094 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:26.165127 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:26.165417 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:26.165418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:26.165420 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:26.165448 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:26.165449 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:26.166680 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:26.166681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:26.167759 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:26.167762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:26.168627 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:26.168628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:26.168687 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:26.168691 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:34:26.168718 (+ 27us) client_negotiation.cc:240] Negotiation successful
0928 02:34:26.168766 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":20,"thread_start_us":67,"threads_started":1}
I20260928 02:34:26.169168 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:26.164994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47310 (local address 127.0.98.2:46121)
0928 02:34:26.165270 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:26.165274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:26.165283 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:34:26.165318 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:26.165319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:26.165333 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:26.165509 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:26.165572 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:26.166588 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:26.167841 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:26.168545 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:26.168969 (+ 424us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:26.169032 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:26.169056 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 02:34:26.169095 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":75,"threads_started":1}
I20260928 02:34:26.169088 1355 tablet_service.cc:1467] Processing CreateTablet for tablet f6e86f17859344ffb442c42bfb7d03c3 (DEFAULT_TABLE table=test-table [id=a269b9ae0d4e4738b0e5e7f0f1dea538]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:26.169415 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6e86f17859344ffb442c42bfb7d03c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:26.169314 1087 tablet_service.cc:1467] Processing CreateTablet for tablet f6e86f17859344ffb442c42bfb7d03c3 (DEFAULT_TABLE table=test-table [id=a269b9ae0d4e4738b0e5e7f0f1dea538]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:26.169773 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6e86f17859344ffb442c42bfb7d03c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:26.169783 1221 tablet_service.cc:1467] Processing CreateTablet for tablet f6e86f17859344ffb442c42bfb7d03c3 (DEFAULT_TABLE table=test-table [id=a269b9ae0d4e4738b0e5e7f0f1dea538]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:26.170081 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6e86f17859344ffb442c42bfb7d03c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:26.172073 1435 tablet_bootstrap.cc:492] T f6e86f17859344ffb442c42bfb7d03c3 P 5a3d698d83be4eee82c4109c37104f41: Bootstrap starting.
I20260928 02:34:26.172713 1435 tablet_bootstrap.cc:654] T f6e86f17859344ffb442c42bfb7d03c3 P 5a3d698d83be4eee82c4109c37104f41: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:26.172966 1437 tablet_bootstrap.cc:492] T f6e86f17859344ffb442c42bfb7d03c3 P 00ee477367894eb5bc4ab5b40a72be22: Bootstrap starting.
I20260928 02:34:26.173163 1436 tablet_bootstrap.cc:492] T f6e86f17859344ffb442c42bfb7d03c3 P fa22b463f48747368debdd4997b74d1b: Bootstrap starting.
I20260928 02:34:26.173246 1435 log.cc:824] T f6e86f17859344ffb442c42bfb7d03c3 P 5a3d698d83be4eee82c4109c37104f41: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:26.173647 1437 tablet_bootstrap.cc:654] T f6e86f17859344ffb442c42bfb7d03c3 P 00ee477367894eb5bc4ab5b40a72be22: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:26.173736 1436 tablet_bootstrap.cc:654] T f6e86f17859344ffb442c42bfb7d03c3 P fa22b463f48747368debdd4997b74d1b: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:26.174014 1435 tablet_bootstrap.cc:492] T f6e86f17859344ffb442c42bfb7d03c3 P 5a3d698d83be4eee82c4109c37104f41: No bootstrap required, opened a new log
I20260928 02:34:26.174088 1435 ts_tablet_manager.cc:1397] T f6e86f17859344ffb442c42bfb7d03c3 P 5a3d698d83be4eee82c4109c37104f41: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:34:26.174093 1437 log.cc:824] T f6e86f17859344ffb442c42bfb7d03c3 P 00ee477367894eb5bc4ab5b40a72be22: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:26.174270 1436 log.cc:824] T f6e86f17859344ffb442c42bfb7d03c3 P fa22b463f48747368debdd4997b74d1b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:26.174752 1437 tablet_bootstrap.cc:492] T f6e86f17859344ffb442c42bfb7d03c3 P 00ee477367894eb5bc4ab5b40a72be22: No bootstrap required, opened a new log
I20260928 02:34:26.174810 1437 ts_tablet_manager.cc:1397] T f6e86f17859344ffb442c42bfb7d03c3 P 00ee477367894eb5bc4ab5b40a72be22: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:34:26.174968 1436 tablet_bootstrap.cc:492] T f6e86f17859344ffb442c42bfb7d03c3 P fa22b463f48747368debdd4997b74d1b: No bootstrap required, opened a new log
I20260928 02:34:26.175021 1436 ts_tablet_manager.cc:1397] T f6e86f17859344ffb442c42bfb7d03c3 P fa22b463f48747368debdd4997b74d1b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:34:26.176222 1435 raft_consensus.cc:348] T f6e86f17859344ffb442c42bfb7d03c3 P 5a3d698d83be4eee82c4109c37104f41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa22b463f48747368debdd4997b74d1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36105 } } peers { permanent_uuid: "00ee477367894eb5bc4ab5b40a72be22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46121 } } peers { permanent_uuid: "5a3d698d83be4eee82c4109c37104f41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40293 } }
I20260928 02:34:26.176299 1437 raft_consensus.cc:348] T f6e86f17859344ffb442c42bfb7d03c3 P 00ee477367894eb5bc4ab5b40a72be22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa22b463f48747368debdd4997b74d1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36105 } } peers { permanent_uuid: "00ee477367894eb5bc4ab5b40a72be22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46121 } } peers { permanent_uuid: "5a3d698d83be4eee82c4109c37104f41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40293 } }
I20260928 02:34:26.176412 1435 raft_consensus.cc:374] T f6e86f17859344ffb442c42bfb7d03c3 P 5a3d698d83be4eee82c4109c37104f41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:26.176425 1437 raft_consensus.cc:374] T f6e86f17859344ffb442c42bfb7d03c3 P 00ee477367894eb5bc4ab5b40a72be22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:26.176450 1435 raft_consensus.cc:729] T f6e86f17859344ffb442c42bfb7d03c3 P 5a3d698d83be4eee82c4109c37104f41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a3d698d83be4eee82c4109c37104f41, State: Initialized, Role: FOLLOWER
I20260928 02:34:26.176456 1437 raft_consensus.cc:729] T f6e86f17859344ffb442c42bfb7d03c3 P 00ee477367894eb5bc4ab5b40a72be22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00ee477367894eb5bc4ab5b40a72be22, State: Initialized, Role: FOLLOWER
I20260928 02:34:26.176577 1437 consensus_queue.cc:260] T f6e86f17859344ffb442c42bfb7d03c3 P 00ee477367894eb5bc4ab5b40a72be22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa22b463f48747368debdd4997b74d1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36105 } } peers { permanent_uuid: "00ee477367894eb5bc4ab5b40a72be22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46121 } } peers { permanent_uuid: "5a3d698d83be4eee82c4109c37104f41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40293 } }
I20260928 02:34:26.176577 1435 consensus_queue.cc:260] T f6e86f17859344ffb442c42bfb7d03c3 P 5a3d698d83be4eee82c4109c37104f41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa22b463f48747368debdd4997b74d1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36105 } } peers { permanent_uuid: "00ee477367894eb5bc4ab5b40a72be22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46121 } } peers { permanent_uuid: "5a3d698d83be4eee82c4109c37104f41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40293 } }
I20260928 02:34:26.176815 1435 ts_tablet_manager.cc:1428] T f6e86f17859344ffb442c42bfb7d03c3 P 5a3d698d83be4eee82c4109c37104f41: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:34:26.176841 1437 ts_tablet_manager.cc:1428] T f6e86f17859344ffb442c42bfb7d03c3 P 00ee477367894eb5bc4ab5b40a72be22: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:34:26.176923 1153 heartbeater.cc:503] Master 127.0.98.62:34723 was elected leader, sending a full tablet report...
I20260928 02:34:26.176942 1287 heartbeater.cc:503] Master 127.0.98.62:34723 was elected leader, sending a full tablet report...
I20260928 02:34:26.177551 1436 raft_consensus.cc:348] T f6e86f17859344ffb442c42bfb7d03c3 P fa22b463f48747368debdd4997b74d1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa22b463f48747368debdd4997b74d1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36105 } } peers { permanent_uuid: "00ee477367894eb5bc4ab5b40a72be22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46121 } } peers { permanent_uuid: "5a3d698d83be4eee82c4109c37104f41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40293 } }
I20260928 02:34:26.177700 1436 raft_consensus.cc:374] T f6e86f17859344ffb442c42bfb7d03c3 P fa22b463f48747368debdd4997b74d1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:26.177734 1436 raft_consensus.cc:729] T f6e86f17859344ffb442c42bfb7d03c3 P fa22b463f48747368debdd4997b74d1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa22b463f48747368debdd4997b74d1b, State: Initialized, Role: FOLLOWER
I20260928 02:34:26.177838 1436 consensus_queue.cc:260] T f6e86f17859344ffb442c42bfb7d03c3 P fa22b463f48747368debdd4997b74d1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa22b463f48747368debdd4997b74d1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36105 } } peers { permanent_uuid: "00ee477367894eb5bc4ab5b40a72be22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46121 } } peers { permanent_uuid: "5a3d698d83be4eee82c4109c37104f41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40293 } }
I20260928 02:34:26.178098 1421 heartbeater.cc:503] Master 127.0.98.62:34723 was elected leader, sending a full tablet report...
I20260928 02:34:26.178082 1436 ts_tablet_manager.cc:1428] T f6e86f17859344ffb442c42bfb7d03c3 P fa22b463f48747368debdd4997b74d1b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:34:26.184111 1443 raft_consensus.cc:482] T f6e86f17859344ffb442c42bfb7d03c3 P fa22b463f48747368debdd4997b74d1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:34:26.184180 1443 raft_consensus.cc:504] T f6e86f17859344ffb442c42bfb7d03c3 P fa22b463f48747368debdd4997b74d1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa22b463f48747368debdd4997b74d1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36105 } } peers { permanent_uuid: "00ee477367894eb5bc4ab5b40a72be22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46121 } } peers { permanent_uuid: "5a3d698d83be4eee82c4109c37104f41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40293 } }
I20260928 02:34:26.184432 1443 leader_election.cc:288] T f6e86f17859344ffb442c42bfb7d03c3 P fa22b463f48747368debdd4997b74d1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00ee477367894eb5bc4ab5b40a72be22 (127.0.98.2:46121), 5a3d698d83be4eee82c4109c37104f41 (127.0.98.1:40293)
I20260928 02:34:26.186800 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:26.184549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46121 (local address 127.0.98.3:46583)
0928 02:34:26.184594 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:26.184604 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:26.184638 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:26.184735 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:26.184735 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:26.184736 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:26.184792 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:26.184793 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:26.185411 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:26.185412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:26.186096 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:26.186098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:26.186590 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:26.186591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:26.186656 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:26.186659 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:34:26.186702 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 02:34:26.186738 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 02:34:26.186798 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:26.184610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46583 (local address 127.0.98.2:46121)
0928 02:34:26.184642 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:26.184644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:26.184666 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:34:26.184685 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:26.184685 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:26.184694 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:26.184734 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:26.184842 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:26.185342 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:26.186168 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:26.186530 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:26.186585 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:26.186594 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:26.186711 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 02:34:26.186741 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 02:34:26.186930 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:26.184615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38431 (local address 127.0.98.1:40293)
0928 02:34:26.184650 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:26.184651 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:26.184751 (+ 100us) server_negotiation.cc:407] Connection header received
0928 02:34:26.184794 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:26.184795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:26.184804 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:26.184845 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:26.184973 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:26.185432 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:26.186289 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:26.186655 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:26.186709 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:26.186718 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:26.186842 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 02:34:26.186874 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:34:26.186944 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:26.184562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40293 (local address 127.0.98.3:38431)
0928 02:34:26.184694 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:26.184702 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:26.184751 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:26.184851 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:26.184852 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:26.184853 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:26.184924 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:26.184925 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:26.185490 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:26.185491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:26.186219 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:26.186221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:26.186717 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:26.186718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:26.186786 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:26.186789 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:34:26.186833 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 02:34:26.186874 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260928 02:34:26.187040 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e86f17859344ffb442c42bfb7d03c3" candidate_uuid: "fa22b463f48747368debdd4997b74d1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00ee477367894eb5bc4ab5b40a72be22" is_pre_election: true
I20260928 02:34:26.187146 1241 raft_consensus.cc:2454] T f6e86f17859344ffb442c42bfb7d03c3 P 00ee477367894eb5bc4ab5b40a72be22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa22b463f48747368debdd4997b74d1b in term 0.
I20260928 02:34:26.187156 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e86f17859344ffb442c42bfb7d03c3" candidate_uuid: "fa22b463f48747368debdd4997b74d1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a3d698d83be4eee82c4109c37104f41" is_pre_election: true
I20260928 02:34:26.187253 1107 raft_consensus.cc:2454] T f6e86f17859344ffb442c42bfb7d03c3 P 5a3d698d83be4eee82c4109c37104f41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa22b463f48747368debdd4997b74d1b in term 0.
I20260928 02:34:26.187341 1308 leader_election.cc:302] T f6e86f17859344ffb442c42bfb7d03c3 P fa22b463f48747368debdd4997b74d1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00ee477367894eb5bc4ab5b40a72be22, fa22b463f48747368debdd4997b74d1b; no voters:
I20260928 02:34:26.187457 1443 raft_consensus.cc:2781] T f6e86f17859344ffb442c42bfb7d03c3 P fa22b463f48747368debdd4997b74d1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:34:26.187508 1443 raft_consensus.cc:482] T f6e86f17859344ffb442c42bfb7d03c3 P fa22b463f48747368debdd4997b74d1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:34:26.187523 1443 raft_consensus.cc:3037] T f6e86f17859344ffb442c42bfb7d03c3 P fa22b463f48747368debdd4997b74d1b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:26.188186 1443 raft_consensus.cc:504] T f6e86f17859344ffb442c42bfb7d03c3 P fa22b463f48747368debdd4997b74d1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa22b463f48747368debdd4997b74d1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36105 } } peers { permanent_uuid: "00ee477367894eb5bc4ab5b40a72be22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46121 } } peers { permanent_uuid: "5a3d698d83be4eee82c4109c37104f41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40293 } }
I20260928 02:34:26.188289 1443 leader_election.cc:288] T f6e86f17859344ffb442c42bfb7d03c3 P fa22b463f48747368debdd4997b74d1b [CANDIDATE]: Term 1 election: Requested vote from peers 00ee477367894eb5bc4ab5b40a72be22 (127.0.98.2:46121), 5a3d698d83be4eee82c4109c37104f41 (127.0.98.1:40293)
I20260928 02:34:26.188419 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e86f17859344ffb442c42bfb7d03c3" candidate_uuid: "fa22b463f48747368debdd4997b74d1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a3d698d83be4eee82c4109c37104f41"
I20260928 02:34:26.188446 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e86f17859344ffb442c42bfb7d03c3" candidate_uuid: "fa22b463f48747368debdd4997b74d1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00ee477367894eb5bc4ab5b40a72be22"
I20260928 02:34:26.188493 1107 raft_consensus.cc:3037] T f6e86f17859344ffb442c42bfb7d03c3 P 5a3d698d83be4eee82c4109c37104f41 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:26.188498 1241 raft_consensus.cc:3037] T f6e86f17859344ffb442c42bfb7d03c3 P 00ee477367894eb5bc4ab5b40a72be22 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:26.189246 1241 raft_consensus.cc:2454] T f6e86f17859344ffb442c42bfb7d03c3 P 00ee477367894eb5bc4ab5b40a72be22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa22b463f48747368debdd4997b74d1b in term 1.
I20260928 02:34:26.189246 1107 raft_consensus.cc:2454] T f6e86f17859344ffb442c42bfb7d03c3 P 5a3d698d83be4eee82c4109c37104f41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa22b463f48747368debdd4997b74d1b in term 1.
I20260928 02:34:26.189405 1310 leader_election.cc:302] T f6e86f17859344ffb442c42bfb7d03c3 P fa22b463f48747368debdd4997b74d1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a3d698d83be4eee82c4109c37104f41, fa22b463f48747368debdd4997b74d1b; no voters:
I20260928 02:34:26.189529 1443 raft_consensus.cc:2781] T f6e86f17859344ffb442c42bfb7d03c3 P fa22b463f48747368debdd4997b74d1b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:26.189647 1443 raft_consensus.cc:686] T f6e86f17859344ffb442c42bfb7d03c3 P fa22b463f48747368debdd4997b74d1b [term 1 LEADER]: Becoming Leader. State: Replica: fa22b463f48747368debdd4997b74d1b, State: Running, Role: LEADER
I20260928 02:34:26.189739 1443 consensus_queue.cc:237] T f6e86f17859344ffb442c42bfb7d03c3 P fa22b463f48747368debdd4997b74d1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa22b463f48747368debdd4997b74d1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36105 } } peers { permanent_uuid: "00ee477367894eb5bc4ab5b40a72be22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46121 } } peers { permanent_uuid: "5a3d698d83be4eee82c4109c37104f41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40293 } }
I20260928 02:34:26.190388 961 catalog_manager.cc:5495] T f6e86f17859344ffb442c42bfb7d03c3 P fa22b463f48747368debdd4997b74d1b reported cstate change: term changed from 0 to 1, leader changed from <none> to fa22b463f48747368debdd4997b74d1b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "fa22b463f48747368debdd4997b74d1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa22b463f48747368debdd4997b74d1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00ee477367894eb5bc4ab5b40a72be22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a3d698d83be4eee82c4109c37104f41" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40293 } health_report { overall_health: UNKNOWN } } }
Sep 28 02:34:26 dist-test-slave-fz47 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 02:34:26.200949 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:26.194794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59046 (local address 127.0.98.1:40293)
0928 02:34:26.194831 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:26.194832 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:26.194942 (+ 110us) server_negotiation.cc:407] Connection header received
0928 02:34:26.194981 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:26.194982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:26.194989 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:26.195032 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:26.195483 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:26.195894 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:26.196551 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:26.196712 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:26.196751 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:26.198788 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:26.198794 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:26.198795 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:26.198817 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:26.200187 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:26.200443 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:26.200445 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:26.200445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:26.200491 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:26.200605 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:26.200605 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:26.200605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:26.200703 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:26.200788 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:26.200842 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 02:34:26.200894 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 28 02:34:26 dist-test-slave-fz47 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562866, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:34:26 dist-test-slave-fz47 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562866, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 02:34:26.221242 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:26.215120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59052 (local address 127.0.98.1:40293)
0928 02:34:26.215164 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:26.215166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:26.215220 (+ 54us) server_negotiation.cc:407] Connection header received
0928 02:34:26.215283 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:26.215284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:26.215293 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:26.215331 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:26.215788 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:26.216239 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:26.216937 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:26.217122 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:26.217174 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:26.219151 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:26.219153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:26.219153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:26.219160 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:26.220296 (+ 1136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:26.220579 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:26.220580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:26.220580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:26.220627 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:26.220780 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:26.220781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:26.220781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:26.220866 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:26.220943 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:26.221091 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 02:34:26.221156 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
Sep 28 02:34:26 dist-test-slave-fz47 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:34:26 dist-test-slave-fz47 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 02:34:26.241310 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:26.235164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59058 (local address 127.0.98.1:40293)
0928 02:34:26.235218 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:26.235220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:26.235284 (+ 64us) server_negotiation.cc:407] Connection header received
0928 02:34:26.235351 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:26.235352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:26.235362 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:26.235407 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:26.235868 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:26.236313 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:26.237045 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:26.237222 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:26.237310 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:26.239249 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:26.239250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:26.239251 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:26.239257 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:26.240386 (+ 1129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:26.240664 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:26.240665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:26.240665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:26.240709 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:26.240865 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:26.240866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:26.240866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:26.240945 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:26.241033 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:26.241131 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 02:34:26.241196 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260928 02:34:26.241703 1067 tablet_service.cc:2908] Rejecting scan request for tablet f6e86f17859344ffb442c42bfb7d03c3: Uninitialized: clean time has not yet been initialized
I20260928 02:34:26.242478 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 1024
W20260928 02:34:26.243160 1288 tablet.cc:2369] T f6e86f17859344ffb442c42bfb7d03c3 P 00ee477367894eb5bc4ab5b40a72be22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:34:26.246769 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 1158
I20260928 02:34:26.250753 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 1292
I20260928 02:34:26.254292 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 930
2026-09-28T02:34:26Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1770 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:34:26 dist-test-slave-fz47 krb5kdc[1459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:34:26 dist-test-slave-fz47 krb5kdc[1459](info): set up 2 sockets
Sep 28 02:34:26 dist-test-slave-fz47 krb5kdc[1459](info): commencing operation
krb5kdc: starting...
W20260928 02:34:27.213240 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260928 02:34:27.333542 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.120s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260928 02:34:27.620390 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.287s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260928 02:34:27.827126 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.207s user 0.000s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/krb5kdc/test-user.keytab.
W20260928 02:34:28.580571 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.753s user 0.000s sys 0.002s
Sep 28 02:34:28 dist-test-slave-fz47 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:34:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:34:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260928 02:34:29.528599 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.928s user 0.000s sys 0.002s
I20260928 02:34:29.528903 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44961
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:29.608006 1472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:29.608197 1472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:29.608217 1472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:29.609365 1472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:34:29.609403 1472 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:34:29.609416 1472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:29.609429 1472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:29.609452 1472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:34:29.609473 1472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:34:29.610936 1472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44961
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:29.611155 1472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:29.611358 1472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:29.615710 1477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:29.615746 1472 server_base.cc:1034] running on GCE node
W20260928 02:34:29.615736 1480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:29.615845 1478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:29.616073 1472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:29.616307 1472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:29.617439 1472 hybrid_clock.cc:648] HybridClock initialized: now 1790562869617424 us; error 29 us; skew 500 ppm
Sep 28 02:34:29 dist-test-slave-fz47 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:29.619624 1472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:34:29.620021 1472 webserver.cc:460] Webserver started at http://127.0.98.62:44887/ using document root <none> and password file <none>
I20260928 02:34:29.620234 1472 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:29.620280 1472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:29.620396 1472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:29.621407 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "c7dda90ee7e44575b410d248043a8c0f"
format_stamp: "Formatted at 2026-09-28 02:34:29 on dist-test-slave-fz47"
server_key: "a84660e5d0b9713f24e2cdfa3d0681f3"
server_key_iv: "fc49762dc6622b76de1d78bbf2af7498"
server_key_version: "encryptionkey@0"
I20260928 02:34:29.621711 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "c7dda90ee7e44575b410d248043a8c0f"
format_stamp: "Formatted at 2026-09-28 02:34:29 on dist-test-slave-fz47"
server_key: "a84660e5d0b9713f24e2cdfa3d0681f3"
server_key_iv: "fc49762dc6622b76de1d78bbf2af7498"
server_key_version: "encryptionkey@0"
I20260928 02:34:29.622845 1472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:34:29.623466 1487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:29.623608 1472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:34:29.623677 1472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "c7dda90ee7e44575b410d248043a8c0f"
format_stamp: "Formatted at 2026-09-28 02:34:29 on dist-test-slave-fz47"
server_key: "a84660e5d0b9713f24e2cdfa3d0681f3"
server_key_iv: "fc49762dc6622b76de1d78bbf2af7498"
server_key_version: "encryptionkey@0"
I20260928 02:34:29.623733 1472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:29.640934 1472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:29.642951 1472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:29.643093 1472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:29.647332 1472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44961
I20260928 02:34:29.647382 1549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44961 every 8 connection(s)
I20260928 02:34:29.647682 1472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:34:29.648260 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:29.650485 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7dda90ee7e44575b410d248043a8c0f: Bootstrap starting.
I20260928 02:34:29.651149 1550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7dda90ee7e44575b410d248043a8c0f: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:29.651407 1550 log.cc:824] T 00000000000000000000000000000000 P c7dda90ee7e44575b410d248043a8c0f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:29.652029 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7dda90ee7e44575b410d248043a8c0f: No bootstrap required, opened a new log
I20260928 02:34:29.652905 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 1472
I20260928 02:34:29.652997 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:34:29.653141 392 external_mini_cluster.cc:1428] Setting key 826c4acffa935b150ec8e7d0172cabd9
I20260928 02:34:29.654438 1550 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7dda90ee7e44575b410d248043a8c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7dda90ee7e44575b410d248043a8c0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44961 } }
I20260928 02:34:29.654578 1550 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7dda90ee7e44575b410d248043a8c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:29.654610 1550 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7dda90ee7e44575b410d248043a8c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7dda90ee7e44575b410d248043a8c0f, State: Initialized, Role: FOLLOWER
I20260928 02:34:29.654767 1550 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7dda90ee7e44575b410d248043a8c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7dda90ee7e44575b410d248043a8c0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44961 } }
I20260928 02:34:29.654846 1550 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7dda90ee7e44575b410d248043a8c0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:34:29.654883 1550 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7dda90ee7e44575b410d248043a8c0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:34:29.654928 1550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7dda90ee7e44575b410d248043a8c0f [term 0 FOLLOWER]: Advancing to term 1
Sep 28 02:34:29 dist-test-slave-fz47 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:29.661759 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:29.653634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57460 (local address 127.0.98.62:44961)
0928 02:34:29.653783 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:29.653791 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:29.653814 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:34:29.654410 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:29.654415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:29.654459 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:29.654600 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:29.655285 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:29.656167 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:29.656845 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:29.657224 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:29.657330 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:29.659279 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:29.659282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:29.659282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:29.659292 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:29.660543 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:29.660844 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:29.660846 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:29.660847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:29.660922 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:29.661045 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:29.661045 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:29.661045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:29.661279 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:29.661385 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:29.661457 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 02:34:29.661592 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260928 02:34:29.676196 1550 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7dda90ee7e44575b410d248043a8c0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7dda90ee7e44575b410d248043a8c0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44961 } }
I20260928 02:34:29.676326 1550 leader_election.cc:302] T 00000000000000000000000000000000 P c7dda90ee7e44575b410d248043a8c0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7dda90ee7e44575b410d248043a8c0f; no voters:
I20260928 02:34:29.676501 1550 leader_election.cc:288] T 00000000000000000000000000000000 P c7dda90ee7e44575b410d248043a8c0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:34:29.676591 1555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7dda90ee7e44575b410d248043a8c0f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:29.676766 1550 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7dda90ee7e44575b410d248043a8c0f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:34:29.676800 1555 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7dda90ee7e44575b410d248043a8c0f [term 1 LEADER]: Becoming Leader. State: Replica: c7dda90ee7e44575b410d248043a8c0f, State: Running, Role: LEADER
I20260928 02:34:29.676993 1555 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7dda90ee7e44575b410d248043a8c0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7dda90ee7e44575b410d248043a8c0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44961 } }
I20260928 02:34:29.677407 1556 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7dda90ee7e44575b410d248043a8c0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7dda90ee7e44575b410d248043a8c0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7dda90ee7e44575b410d248043a8c0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44961 } } }
I20260928 02:34:29.677489 1556 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7dda90ee7e44575b410d248043a8c0f [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:29.677392 1557 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7dda90ee7e44575b410d248043a8c0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7dda90ee7e44575b410d248043a8c0f. Latest consensus state: current_term: 1 leader_uuid: "c7dda90ee7e44575b410d248043a8c0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7dda90ee7e44575b410d248043a8c0f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44961 } } }
I20260928 02:34:29.677668 1557 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7dda90ee7e44575b410d248043a8c0f [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:29.677763 1564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:34:29.678601 1564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:34:29.680444 1564 catalog_manager.cc:1347] Generated new cluster ID: 2cba028618a7462facc3ccede41ddc68
I20260928 02:34:29.680490 1564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:34:29.706529 1564 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:34:29.707206 1564 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:34:29.710438 1564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7dda90ee7e44575b410d248043a8c0f: Generated new TSK 0
I20260928 02:34:29.710614 1564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260928 02:34:30.641031 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 0.925s user 0.002s sys 0.001s
I20260928 02:34:30.641358 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44961
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:30.721526 1576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:30.721704 1576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:30.721724 1576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:30.722856 1576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:30.722894 1576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:30.722913 1576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:34:30.722942 1576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:34:30.724382 1576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:30.724627 1576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:30.724817 1576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:30.729506 1582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:30.729524 1584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:30.729537 1576 server_base.cc:1034] running on GCE node
W20260928 02:34:30.729511 1581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:30.729888 1576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:30.730072 1576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:30.731199 1576 hybrid_clock.cc:648] HybridClock initialized: now 1790562870731187 us; error 28 us; skew 500 ppm
Sep 28 02:34:30 dist-test-slave-fz47 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:30.738615 1576 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 02:34:30.738993 1576 webserver.cc:460] Webserver started at http://127.0.98.1:45091/ using document root <none> and password file <none>
I20260928 02:34:30.739216 1576 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:30.739264 1576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:30.739384 1576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:30.740406 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-0/data/instance:
uuid: "cc80b62d7fdb4fa0902c077dd91948d5"
format_stamp: "Formatted at 2026-09-28 02:34:30 on dist-test-slave-fz47"
server_key: "0b7a1c430abeb823fa85a28df979a8dc"
server_key_iv: "fa458ebd40910d4ebe9925286283845b"
server_key_version: "encryptionkey@0"
I20260928 02:34:30.740720 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cc80b62d7fdb4fa0902c077dd91948d5"
format_stamp: "Formatted at 2026-09-28 02:34:30 on dist-test-slave-fz47"
server_key: "0b7a1c430abeb823fa85a28df979a8dc"
server_key_iv: "fa458ebd40910d4ebe9925286283845b"
server_key_version: "encryptionkey@0"
I20260928 02:34:30.741931 1576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:34:30.742570 1591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:30.742717 1576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:30.742787 1576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "cc80b62d7fdb4fa0902c077dd91948d5"
format_stamp: "Formatted at 2026-09-28 02:34:30 on dist-test-slave-fz47"
server_key: "0b7a1c430abeb823fa85a28df979a8dc"
server_key_iv: "fa458ebd40910d4ebe9925286283845b"
server_key_version: "encryptionkey@0"
I20260928 02:34:30.742843 1576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:30.769402 1576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:30.771142 1576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:30.771270 1576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:30.771739 1576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:30.771776 1576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:30.771811 1576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:30.771842 1576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:34:30 dist-test-slave-fz47 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:30.778517 1576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33325
I20260928 02:34:30.778584 1707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33325 every 8 connection(s)
I20260928 02:34:30.778928 1576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
I20260928 02:34:30.781160 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:30.772229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58615 (local address 127.0.98.62:44961)
0928 02:34:30.772333 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:30.772336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:30.772532 (+ 196us) server_negotiation.cc:407] Connection header received
0928 02:34:30.773079 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:30.773081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:30.773096 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:30.773168 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:30.773883 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:30.774423 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:30.775395 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:30.775636 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:30.775694 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:30.778785 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:30.778787 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:30.778787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:30.778793 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:30.780081 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:30.780430 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:30.780432 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:30.780432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:30.780492 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:30.780699 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:30.780699 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:30.780700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:30.780802 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:30.780847 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:30.781032 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 02:34:30.781093 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":51,"threads_started":1}
I20260928 02:34:30.781322 1601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:30.772252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44961 (local address 127.0.98.1:58615)
0928 02:34:30.772392 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:30.772445 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:30.772937 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:30.773282 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:30.773286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:30.773576 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:30.773783 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:30.773788 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:30.774514 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:30.774515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:30.775311 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:30.775313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:30.775709 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:30.775710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:30.775835 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:30.776314 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:30.776324 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:30.778708 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:30.780147 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:30.780153 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:30.780155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:30.780349 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:30.780549 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:30.780550 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:30.780550 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:30.780654 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:30.780844 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:30.780847 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:30.780949 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 02:34:30.781020 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 02:34:30.781130 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":72,"threads_started":1}
I20260928 02:34:30.781785 1708 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44961
I20260928 02:34:30.781857 1708 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:30.782008 1708 heartbeater.cc:511] Master 127.0.98.62:44961 requested a full tablet report, sending...
I20260928 02:34:30.782521 1504 ts_manager.cc:194] Registered new tserver with Master: cc80b62d7fdb4fa0902c077dd91948d5 (127.0.98.1:33325)
I20260928 02:34:30.783142 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58615
I20260928 02:34:30.785701 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 1576
I20260928 02:34:30.785784 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-0/wal/instance
I20260928 02:34:30.785945 392 external_mini_cluster.cc:1428] Setting key 2150366920949209d0af88a7d35382f6
I20260928 02:34:30.787962 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:30.782792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44961 (local address 127.0.98.1:34721)
0928 02:34:30.782898 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:30.782907 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:30.782960 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:30.783060 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:30.783060 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:30.783221 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:30.783275 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:30.783276 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:30.784096 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:30.784097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:30.784591 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:30.784592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:30.784943 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:30.784944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:30.785005 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:30.785394 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:30.785397 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:30.785643 (+ 246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:30.787120 (+ 1477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:30.787120 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:30.787121 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:30.787267 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:30.787425 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:30.787426 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:30.787426 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:30.787480 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:30.787722 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:30.787723 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:30.787763 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 02:34:30.787823 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 02:34:30.787894 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":63,"threads_started":1}
I20260928 02:34:30.788009 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:30.782733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34721 (local address 127.0.98.62:44961)
0928 02:34:30.782767 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:30.782768 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:30.782962 (+ 194us) server_negotiation.cc:407] Connection header received
0928 02:34:30.783009 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:30.783010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:30.783019 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:30.783058 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:30.783339 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:30.783997 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:30.784657 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:30.784886 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:30.784936 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:30.785714 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:30.785715 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:30.785715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:30.785722 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:30.787040 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:30.787323 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:30.787324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:30.787324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:30.787375 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:30.787528 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:30.787529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:30.787529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:30.787657 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:30.787719 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:30.787854 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 02:34:30.787927 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260928 02:34:31.424070 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 0.637s user 0.000s sys 0.002s
I20260928 02:34:31.424418 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44961
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:31.503198 1717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:31.503400 1717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:31.503424 1717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:31.504513 1717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:31.504549 1717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:31.504568 1717 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:34:31.504583 1717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:34:31.506623 1717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:31.506850 1717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:31.507076 1717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:31.512203 1723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:31.512221 1722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:31.512207 1725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:31.512435 1717 server_base.cc:1034] running on GCE node
I20260928 02:34:31.512683 1717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:31.512912 1717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:31.514041 1717 hybrid_clock.cc:648] HybridClock initialized: now 1790562871514031 us; error 33 us; skew 500 ppm
Sep 28 02:34:31 dist-test-slave-fz47 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:31.516345 1717 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 02:34:31.516759 1717 webserver.cc:460] Webserver started at http://127.0.98.2:46657/ using document root <none> and password file <none>
I20260928 02:34:31.516940 1717 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:31.516980 1717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:31.517081 1717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:31.518102 1717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-1/data/instance:
uuid: "03510eb2c310483fa77baf78af35dbef"
format_stamp: "Formatted at 2026-09-28 02:34:31 on dist-test-slave-fz47"
server_key: "596f2b7df0f13319a4cdfab9ea321e0d"
server_key_iv: "42b0a82bbe04d988cabecd8ce2a911b6"
server_key_version: "encryptionkey@0"
I20260928 02:34:31.518419 1717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-1/wal/instance:
uuid: "03510eb2c310483fa77baf78af35dbef"
format_stamp: "Formatted at 2026-09-28 02:34:31 on dist-test-slave-fz47"
server_key: "596f2b7df0f13319a4cdfab9ea321e0d"
server_key_iv: "42b0a82bbe04d988cabecd8ce2a911b6"
server_key_version: "encryptionkey@0"
I20260928 02:34:31.519620 1717 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:34:31.520236 1732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:31.520381 1717 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:31.520478 1717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "03510eb2c310483fa77baf78af35dbef"
format_stamp: "Formatted at 2026-09-28 02:34:31 on dist-test-slave-fz47"
server_key: "596f2b7df0f13319a4cdfab9ea321e0d"
server_key_iv: "42b0a82bbe04d988cabecd8ce2a911b6"
server_key_version: "encryptionkey@0"
I20260928 02:34:31.520531 1717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:31.540535 1717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:31.542361 1717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:31.542496 1717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:31.542960 1717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:31.543012 1717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:31.543051 1717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:31.543067 1717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:31.549854 1717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38343
I20260928 02:34:31.549939 1848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38343 every 8 connection(s)
Sep 28 02:34:31 dist-test-slave-fz47 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:31.550206 1717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
I20260928 02:34:31.552961 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:31.543552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46145 (local address 127.0.98.62:44961)
0928 02:34:31.543680 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:31.543682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:31.543896 (+ 214us) server_negotiation.cc:407] Connection header received
0928 02:34:31.544481 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:31.544484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:31.544497 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:31.544563 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:31.545465 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:31.546237 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:31.547727 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:31.548035 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:31.548093 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:31.550713 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:31.550715 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:31.550715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:31.550722 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:31.551846 (+ 1124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:31.552176 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:31.552176 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:31.552177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:31.552224 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:31.552402 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:31.552402 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:31.552403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:31.552518 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:31.552567 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:31.552827 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 02:34:31.552892 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260928 02:34:31.553162 1742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:31.543555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44961 (local address 127.0.98.2:46145)
0928 02:34:31.543760 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:31.543812 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:31.544375 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:31.544668 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:31.544674 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:31.545058 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:31.545382 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:31.545388 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:31.546329 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:31.546330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:31.547644 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:31.547647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:31.548111 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:31.548112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:31.548256 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:31.548768 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:31.548785 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:31.550636 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:31.551917 (+ 1281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:31.551922 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:31.551924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:31.552120 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:31.552267 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:31.552267 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:31.552267 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:31.552351 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:31.552593 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:31.552599 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:31.552715 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 02:34:31.552806 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 02:34:31.552939 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":77,"threads_started":1}
I20260928 02:34:31.553581 1849 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44961
I20260928 02:34:31.553661 1849 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:31.553833 1849 heartbeater.cc:511] Master 127.0.98.62:44961 requested a full tablet report, sending...
I20260928 02:34:31.554214 1503 ts_manager.cc:194] Registered new tserver with Master: 03510eb2c310483fa77baf78af35dbef (127.0.98.2:38343)
I20260928 02:34:31.554571 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46145
I20260928 02:34:31.558724 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 1717
I20260928 02:34:31.558816 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-1/wal/instance
I20260928 02:34:31.559000 392 external_mini_cluster.cc:1428] Setting key 73450157dadb19338ee7d093c0183427
I20260928 02:34:31.559331 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:31.554582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50321 (local address 127.0.98.62:44961)
0928 02:34:31.554621 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:31.554623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:31.554691 (+ 68us) server_negotiation.cc:407] Connection header received
0928 02:34:31.554762 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:31.554763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:31.554772 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:31.554834 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:31.555165 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:31.555620 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:31.556247 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:31.556441 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:31.556486 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:31.557149 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:31.557150 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:31.557150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:31.557156 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:31.558209 (+ 1053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:31.558473 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:31.558474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:31.558474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:31.558518 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:31.558763 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:31.558764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:31.558765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:31.558941 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:31.559008 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:31.559168 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 02:34:31.559253 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:34:31.559387 1855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:31.554519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44961 (local address 127.0.98.2:50321)
0928 02:34:31.554634 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:31.554643 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:31.554715 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:31.554842 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:31.554842 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:31.555043 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:31.555097 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:31.555098 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:31.555695 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:31.555696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:31.556182 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:31.556183 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:31.556489 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:31.556490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:31.556529 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:31.556839 (+ 310us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:31.556842 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:31.557085 (+ 243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:31.558280 (+ 1195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:31.558281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:31.558281 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:31.558424 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:31.558575 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:31.558576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:31.558576 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:31.558693 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:31.559018 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:31.559019 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:31.559055 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 02:34:31.559143 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 02:34:31.559286 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":66,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260928 02:34:31.784070 1708 heartbeater.cc:503] Master 127.0.98.62:44961 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260928 02:34:32.285989 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.726s user 0.000s sys 0.003s
I20260928 02:34:32.286302 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44961
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:32.363992 1858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:32.364188 1858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:32.364217 1858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:32.365322 1858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:32.365365 1858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:32.365391 1858 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:34:32.365412 1858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:34:32.366842 1858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:32.367060 1858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:32.367257 1858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:32.371985 1866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:32.372008 1863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:32.372068 1858 server_base.cc:1034] running on GCE node
W20260928 02:34:32.371985 1864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:32.372386 1858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:32.372624 1858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:32.373791 1858 hybrid_clock.cc:648] HybridClock initialized: now 1790562872373780 us; error 29 us; skew 500 ppm
Sep 28 02:34:32 dist-test-slave-fz47 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:32.376385 1858 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 02:34:32.376754 1858 webserver.cc:460] Webserver started at http://127.0.98.3:42371/ using document root <none> and password file <none>
I20260928 02:34:32.376999 1858 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:32.377053 1858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:32.377144 1858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:32.378197 1858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-2/data/instance:
uuid: "cfec3cfacb084f148da0bdc30378ea79"
format_stamp: "Formatted at 2026-09-28 02:34:32 on dist-test-slave-fz47"
server_key: "4695d3f438cbd6b6577128a95e4f5202"
server_key_iv: "270b80b50aff68757327363e8cde187a"
server_key_version: "encryptionkey@0"
I20260928 02:34:32.378494 1858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cfec3cfacb084f148da0bdc30378ea79"
format_stamp: "Formatted at 2026-09-28 02:34:32 on dist-test-slave-fz47"
server_key: "4695d3f438cbd6b6577128a95e4f5202"
server_key_iv: "270b80b50aff68757327363e8cde187a"
server_key_version: "encryptionkey@0"
I20260928 02:34:32.379642 1858 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:34:32.380306 1873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:32.380533 1858 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:34:32.380604 1858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "cfec3cfacb084f148da0bdc30378ea79"
format_stamp: "Formatted at 2026-09-28 02:34:32 on dist-test-slave-fz47"
server_key: "4695d3f438cbd6b6577128a95e4f5202"
server_key_iv: "270b80b50aff68757327363e8cde187a"
server_key_version: "encryptionkey@0"
I20260928 02:34:32.380653 1858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:32.402680 1858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:32.404595 1858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:32.404703 1858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:32.405176 1858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:32.405215 1858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:32.405243 1858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:32.405282 1858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:32.411577 1858 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46255
I20260928 02:34:32.411667 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46255 every 8 connection(s)
Sep 28 02:34:32 dist-test-slave-fz47 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:32.411964 1858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
I20260928 02:34:32.415118 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:32.405697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60785 (local address 127.0.98.62:44961)
0928 02:34:32.405801 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:32.405803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:32.405977 (+ 174us) server_negotiation.cc:407] Connection header received
0928 02:34:32.406542 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:32.406544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:32.406560 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:32.406632 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:32.407499 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.408281 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.409691 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.409907 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.409963 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:32.412455 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:32.412456 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:32.412457 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:32.412464 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:32.413851 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:32.414212 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:32.414213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:32.414214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:32.414270 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:32.414489 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:32.414490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:32.414490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:32.414604 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:32.414661 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:32.414926 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 02:34:32.415024 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":47,"threads_started":1}
I20260928 02:34:32.415270 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:32.405695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44961 (local address 127.0.98.3:60785)
0928 02:34:32.405852 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:32.405897 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:32.406394 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:32.406728 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:32.406732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:32.407071 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:32.407395 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:32.407403 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.408385 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.408386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:32.409605 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:32.409607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.409975 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.409976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:32.410103 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:32.410641 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:32.410651 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:32.412361 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:32.413931 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:32.413938 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:32.413940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:32.414146 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:32.414326 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:32.414327 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:32.414327 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:32.414434 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:32.414660 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:32.414665 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:32.414799 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 02:34:32.414893 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 02:34:32.415053 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
I20260928 02:34:32.415709 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44961
I20260928 02:34:32.415786 1990 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:32.415962 1990 heartbeater.cc:511] Master 127.0.98.62:44961 requested a full tablet report, sending...
I20260928 02:34:32.416342 1503 ts_manager.cc:194] Registered new tserver with Master: cfec3cfacb084f148da0bdc30378ea79 (127.0.98.3:46255)
I20260928 02:34:32.416718 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60785
I20260928 02:34:32.420616 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 1858
I20260928 02:34:32.420696 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-2/wal/instance
I20260928 02:34:32.420848 392 external_mini_cluster.cc:1428] Setting key 6cbff9de12e1fc9c7d5b028374657828
I20260928 02:34:32.421665 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:34:32.421902 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:32.416729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48265 (local address 127.0.98.62:44961)
0928 02:34:32.416768 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:32.416770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:32.416841 (+ 71us) server_negotiation.cc:407] Connection header received
0928 02:34:32.416902 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:32.416903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:32.416912 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:32.416973 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:32.417305 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.417832 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.418498 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.418683 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.418735 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:32.419550 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:32.419551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:32.419551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:32.419557 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:32.420808 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:32.421166 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:32.421167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:32.421168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:32.421227 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:32.421437 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:32.421438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:32.421438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:32.421564 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:32.421628 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:32.421770 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 02:34:32.421842 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:34:32.421948 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:32.416658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44961 (local address 127.0.98.3:48265)
0928 02:34:32.416776 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:32.416787 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:32.416855 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:32.416974 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:32.416975 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:32.417153 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:32.417216 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:32.417218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.417912 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.417913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:32.418429 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:32.418431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.418737 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.418738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:32.418782 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:32.419189 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:32.419193 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:32.419476 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:32.420894 (+ 1418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:32.420895 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:32.420895 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:32.421096 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:32.421309 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:32.421310 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:32.421310 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:32.421374 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:32.421627 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:32.421628 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:32.421672 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 02:34:32.421738 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 02:34:32.421854 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":62,"threads_started":1}
Sep 28 02:34:32 dist-test-slave-fz47 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562872, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:34:32 dist-test-slave-fz47 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562872, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:32.442764 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:32.436609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48970 (local address 127.0.98.62:44961)
0928 02:34:32.436654 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:32.436656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:32.436714 (+ 58us) server_negotiation.cc:407] Connection header received
0928 02:34:32.436765 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:32.436766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:32.436774 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:32.436842 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:32.437345 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.437854 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.438539 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.438725 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.438774 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:32.440674 (+ 1900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:32.440676 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:32.440676 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:32.440683 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:32.441954 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:32.442234 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:32.442235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:32.442235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:32.442282 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:32.442407 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:32.442407 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:32.442408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:32.442512 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:32.442586 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:32.442651 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 02:34:32.442701 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 02:34:32.444212 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:34:32.445787 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:34:32.451930 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:32.448084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36412 (local address 127.0.98.2:38343)
0928 02:34:32.448189 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:32.448194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:32.448364 (+ 170us) server_negotiation.cc:407] Connection header received
0928 02:34:32.448494 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:32.448496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:32.448512 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:32.448557 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:32.448855 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.449583 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.451041 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.451551 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.451610 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:32.451668 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:32.451806 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 02:34:32.451846 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":57,"threads_started":1}
I20260928 02:34:32.451930 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:32.447969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38343 (local address 127.0.0.1:36412)
0928 02:34:32.448290 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:32.448300 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:32.448400 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:32.448641 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:32.448643 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:32.448648 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:32.448808 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:32.448809 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.449649 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.449650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:32.450952 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:32.450954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.451627 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.451628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:32.451703 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:32.451715 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 02:34:32.451778 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 02:34:32.451843 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":122,"thread_start_us":83,"threads_started":1}
I20260928 02:34:32.451936 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:32.448182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33325 (local address 127.0.0.1:33162)
0928 02:34:32.448387 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:32.448395 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:32.448436 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:32.448641 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:32.448643 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:32.448645 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:32.448746 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:32.448749 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.450220 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.450221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:32.451049 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:32.451052 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.451633 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.451634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:32.451706 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:32.451716 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 02:34:32.451783 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 02:34:32.451848 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":38,"thread_start_us":56,"threads_started":1}
I20260928 02:34:32.452499 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:32.448371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33162 (local address 127.0.98.1:33325)
0928 02:34:32.448485 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:32.448489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:32.448498 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:34:32.448531 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:32.448533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:32.448548 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:32.448625 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:32.449326 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.450155 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.451118 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.451559 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.451623 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:32.451678 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:32.452383 (+ 705us) server_negotiation.cc:299] Negotiation successful
0928 02:34:32.452427 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":68,"threads_started":1}
I20260928 02:34:32.452615 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:32.447962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46255 (local address 127.0.0.1:49918)
0928 02:34:32.448232 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:32.448271 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:32.448349 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:32.448661 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:32.448662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:32.448664 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:32.448746 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:32.448748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.449825 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.449826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:32.450951 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:32.450954 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.452400 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.452401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:32.452460 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:32.452464 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:34:32.452516 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 02:34:32.452551 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":31,"thread_start_us":77,"threads_started":1}
I20260928 02:34:32.452641 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:32.448050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49918 (local address 127.0.98.3:46255)
0928 02:34:32.448269 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:32.448273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:32.448518 (+ 245us) server_negotiation.cc:407] Connection header received
0928 02:34:32.448554 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:32.448556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:32.448572 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:32.448723 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:32.448809 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.449750 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.451702 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.452331 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.452390 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:32.452453 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:32.452527 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 02:34:32.452557 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":76,"threads_started":1}
I20260928 02:34:32.452883 1783 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6f92b52dfc4252bdec594ec4987490 (DEFAULT_TABLE table=test-table [id=0689e201f1564e35827688c4938cfb80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:32.453311 1783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6f92b52dfc4252bdec594ec4987490. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:32.453238 1642 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6f92b52dfc4252bdec594ec4987490 (DEFAULT_TABLE table=test-table [id=0689e201f1564e35827688c4938cfb80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:32.453480 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6f92b52dfc4252bdec594ec4987490 (DEFAULT_TABLE table=test-table [id=0689e201f1564e35827688c4938cfb80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:32.453579 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6f92b52dfc4252bdec594ec4987490. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:32.453770 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6f92b52dfc4252bdec594ec4987490. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:32.455680 2014 tablet_bootstrap.cc:492] T 3f6f92b52dfc4252bdec594ec4987490 P 03510eb2c310483fa77baf78af35dbef: Bootstrap starting.
I20260928 02:34:32.456240 2014 tablet_bootstrap.cc:654] T 3f6f92b52dfc4252bdec594ec4987490 P 03510eb2c310483fa77baf78af35dbef: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:32.456514 2014 log.cc:824] T 3f6f92b52dfc4252bdec594ec4987490 P 03510eb2c310483fa77baf78af35dbef: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:32.456588 2015 tablet_bootstrap.cc:492] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5: Bootstrap starting.
I20260928 02:34:32.457165 2017 tablet_bootstrap.cc:492] T 3f6f92b52dfc4252bdec594ec4987490 P cfec3cfacb084f148da0bdc30378ea79: Bootstrap starting.
I20260928 02:34:32.457191 2015 tablet_bootstrap.cc:654] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:32.457242 2014 tablet_bootstrap.cc:492] T 3f6f92b52dfc4252bdec594ec4987490 P 03510eb2c310483fa77baf78af35dbef: No bootstrap required, opened a new log
I20260928 02:34:32.457330 2014 ts_tablet_manager.cc:1397] T 3f6f92b52dfc4252bdec594ec4987490 P 03510eb2c310483fa77baf78af35dbef: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:34:32.457510 2015 log.cc:824] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:32.457659 2017 tablet_bootstrap.cc:654] T 3f6f92b52dfc4252bdec594ec4987490 P cfec3cfacb084f148da0bdc30378ea79: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:32.457937 2017 log.cc:824] T 3f6f92b52dfc4252bdec594ec4987490 P cfec3cfacb084f148da0bdc30378ea79: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:32.458211 2015 tablet_bootstrap.cc:492] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5: No bootstrap required, opened a new log
I20260928 02:34:32.458266 2015 ts_tablet_manager.cc:1397] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 02:34:32.458487 2017 tablet_bootstrap.cc:492] T 3f6f92b52dfc4252bdec594ec4987490 P cfec3cfacb084f148da0bdc30378ea79: No bootstrap required, opened a new log
I20260928 02:34:32.458534 2017 ts_tablet_manager.cc:1397] T 3f6f92b52dfc4252bdec594ec4987490 P cfec3cfacb084f148da0bdc30378ea79: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:34:32.459717 2014 raft_consensus.cc:348] T 3f6f92b52dfc4252bdec594ec4987490 P 03510eb2c310483fa77baf78af35dbef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } }
I20260928 02:34:32.459877 2014 raft_consensus.cc:374] T 3f6f92b52dfc4252bdec594ec4987490 P 03510eb2c310483fa77baf78af35dbef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:32.459911 2014 raft_consensus.cc:729] T 3f6f92b52dfc4252bdec594ec4987490 P 03510eb2c310483fa77baf78af35dbef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03510eb2c310483fa77baf78af35dbef, State: Initialized, Role: FOLLOWER
I20260928 02:34:32.459877 2017 raft_consensus.cc:348] T 3f6f92b52dfc4252bdec594ec4987490 P cfec3cfacb084f148da0bdc30378ea79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } }
I20260928 02:34:32.459983 2017 raft_consensus.cc:374] T 3f6f92b52dfc4252bdec594ec4987490 P cfec3cfacb084f148da0bdc30378ea79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:32.460007 2017 raft_consensus.cc:729] T 3f6f92b52dfc4252bdec594ec4987490 P cfec3cfacb084f148da0bdc30378ea79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfec3cfacb084f148da0bdc30378ea79, State: Initialized, Role: FOLLOWER
I20260928 02:34:32.460024 2014 consensus_queue.cc:260] T 3f6f92b52dfc4252bdec594ec4987490 P 03510eb2c310483fa77baf78af35dbef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } }
I20260928 02:34:32.460101 2017 consensus_queue.cc:260] T 3f6f92b52dfc4252bdec594ec4987490 P cfec3cfacb084f148da0bdc30378ea79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } }
I20260928 02:34:32.460229 2014 ts_tablet_manager.cc:1428] T 3f6f92b52dfc4252bdec594ec4987490 P 03510eb2c310483fa77baf78af35dbef: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 02:34:32.460271 1849 heartbeater.cc:503] Master 127.0.98.62:44961 was elected leader, sending a full tablet report...
I20260928 02:34:32.460320 2017 ts_tablet_manager.cc:1428] T 3f6f92b52dfc4252bdec594ec4987490 P cfec3cfacb084f148da0bdc30378ea79: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:34:32.460422 1990 heartbeater.cc:503] Master 127.0.98.62:44961 was elected leader, sending a full tablet report...
I20260928 02:34:32.460852 2015 raft_consensus.cc:348] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } }
I20260928 02:34:32.460973 2015 raft_consensus.cc:374] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:32.461002 2015 raft_consensus.cc:729] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc80b62d7fdb4fa0902c077dd91948d5, State: Initialized, Role: FOLLOWER
I20260928 02:34:32.461088 2015 consensus_queue.cc:260] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } }
I20260928 02:34:32.461308 2015 ts_tablet_manager.cc:1428] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 02:34:32.524310 2022 raft_consensus.cc:482] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:34:32.524507 2022 raft_consensus.cc:504] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } }
I20260928 02:34:32.524868 2022 leader_election.cc:288] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfec3cfacb084f148da0bdc30378ea79 (127.0.98.3:46255), 03510eb2c310483fa77baf78af35dbef (127.0.98.2:38343)
I20260928 02:34:32.528687 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:32.525015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46255 (local address 127.0.98.1:46165)
0928 02:34:32.525210 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:32.525221 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:32.525319 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:32.525476 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:32.525477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:32.525479 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:32.525574 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:32.525576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.526254 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.526255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:32.527343 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:32.527347 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.528446 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.528447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:32.528523 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:32.528527 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:34:32.528574 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 02:34:32.528605 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":112,"threads_started":1}
I20260928 02:34:32.528699 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:32.525134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46165 (local address 127.0.98.3:46255)
0928 02:34:32.525229 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:32.525231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:32.525337 (+ 106us) server_negotiation.cc:407] Connection header received
0928 02:34:32.525403 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:32.525403 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:32.525415 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:32.525459 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:32.525648 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.526179 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.527437 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.528317 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.528430 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:32.528445 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:32.528589 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 02:34:32.528631 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:34:32.528793 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:32.525115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42281 (local address 127.0.98.2:38343)
0928 02:34:32.525241 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:32.525243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:32.525303 (+ 60us) server_negotiation.cc:407] Connection header received
0928 02:34:32.525393 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:32.525395 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:32.525417 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:32.525507 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:32.525660 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.526578 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.527909 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.528413 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.528502 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:32.528516 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:32.528681 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 02:34:32.528731 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 02:34:32.528785 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:32.525034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38343 (local address 127.0.98.1:42281)
0928 02:34:32.525222 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:32.525232 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:32.525310 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:32.525516 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:32.525517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:32.525518 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:32.525588 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:32.525589 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.526708 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.526709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:32.527805 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:32.527809 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.528523 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.528524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:32.528602 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:32.528606 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:34:32.528669 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 02:34:32.528706 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":97,"threads_started":1}
I20260928 02:34:32.529215 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6f92b52dfc4252bdec594ec4987490" candidate_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfec3cfacb084f148da0bdc30378ea79" is_pre_election: true
I20260928 02:34:32.529223 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6f92b52dfc4252bdec594ec4987490" candidate_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03510eb2c310483fa77baf78af35dbef" is_pre_election: true
I20260928 02:34:32.529408 1944 raft_consensus.cc:2454] T 3f6f92b52dfc4252bdec594ec4987490 P cfec3cfacb084f148da0bdc30378ea79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc80b62d7fdb4fa0902c077dd91948d5 in term 0.
I20260928 02:34:32.529431 1803 raft_consensus.cc:2454] T 3f6f92b52dfc4252bdec594ec4987490 P 03510eb2c310483fa77baf78af35dbef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc80b62d7fdb4fa0902c077dd91948d5 in term 0.
I20260928 02:34:32.529633 1595 leader_election.cc:302] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03510eb2c310483fa77baf78af35dbef, cc80b62d7fdb4fa0902c077dd91948d5; no voters:
I20260928 02:34:32.529760 2022 raft_consensus.cc:2781] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:34:32.529837 2022 raft_consensus.cc:482] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:34:32.529855 2022 raft_consensus.cc:3037] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5 [term 0 FOLLOWER]: Advancing to term 1
W20260928 02:34:32.530139 1709 tablet.cc:2369] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:34:32.530682 2022 raft_consensus.cc:504] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } }
I20260928 02:34:32.530798 2022 leader_election.cc:288] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5 [CANDIDATE]: Term 1 election: Requested vote from peers cfec3cfacb084f148da0bdc30378ea79 (127.0.98.3:46255), 03510eb2c310483fa77baf78af35dbef (127.0.98.2:38343)
I20260928 02:34:32.530941 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6f92b52dfc4252bdec594ec4987490" candidate_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfec3cfacb084f148da0bdc30378ea79"
I20260928 02:34:32.531023 1944 raft_consensus.cc:3037] T 3f6f92b52dfc4252bdec594ec4987490 P cfec3cfacb084f148da0bdc30378ea79 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:32.531026 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6f92b52dfc4252bdec594ec4987490" candidate_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03510eb2c310483fa77baf78af35dbef"
I20260928 02:34:32.531090 1803 raft_consensus.cc:3037] T 3f6f92b52dfc4252bdec594ec4987490 P 03510eb2c310483fa77baf78af35dbef [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:32.531840 1944 raft_consensus.cc:2454] T 3f6f92b52dfc4252bdec594ec4987490 P cfec3cfacb084f148da0bdc30378ea79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc80b62d7fdb4fa0902c077dd91948d5 in term 1.
I20260928 02:34:32.531857 1803 raft_consensus.cc:2454] T 3f6f92b52dfc4252bdec594ec4987490 P 03510eb2c310483fa77baf78af35dbef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc80b62d7fdb4fa0902c077dd91948d5 in term 1.
I20260928 02:34:32.531989 1595 leader_election.cc:302] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03510eb2c310483fa77baf78af35dbef, cc80b62d7fdb4fa0902c077dd91948d5; no voters:
I20260928 02:34:32.532104 2022 raft_consensus.cc:2781] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:32.532243 2022 raft_consensus.cc:686] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5 [term 1 LEADER]: Becoming Leader. State: Replica: cc80b62d7fdb4fa0902c077dd91948d5, State: Running, Role: LEADER
I20260928 02:34:32.532336 2022 consensus_queue.cc:237] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } }
I20260928 02:34:32.533113 1503 catalog_manager.cc:5495] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc80b62d7fdb4fa0902c077dd91948d5 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } health_report { overall_health: HEALTHY } } }
I20260928 02:34:32.547128 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:32.544827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33176 (local address 127.0.98.1:33325)
0928 02:34:32.544881 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:32.544883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:32.544890 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:34:32.544954 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:32.544954 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:32.544970 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:32.545023 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:34:32.545137 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.545704 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.546476 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.546656 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.546714 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:32.546869 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:34:32.546954 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:34:32.547003 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:32.547032 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 02:34:32.547046 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 02:34:32.548738 1944 raft_consensus.cc:1261] T 3f6f92b52dfc4252bdec594ec4987490 P cfec3cfacb084f148da0bdc30378ea79 [term 1 FOLLOWER]: Refusing update from remote peer cc80b62d7fdb4fa0902c077dd91948d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:32.548738 1803 raft_consensus.cc:1261] T 3f6f92b52dfc4252bdec594ec4987490 P 03510eb2c310483fa77baf78af35dbef [term 1 FOLLOWER]: Refusing update from remote peer cc80b62d7fdb4fa0902c077dd91948d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:32.549068 2025 consensus_queue.cc:1035] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:32.549198 2022 consensus_queue.cc:1035] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:34:32.551246 1850 tablet.cc:2369] T 3f6f92b52dfc4252bdec594ec4987490 P 03510eb2c310483fa77baf78af35dbef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:34:32.552541 2029 mvcc.cc:205] Tried to move back new op lower bound from 7334145525957238784 to 7334145525893107712. Current Snapshot: MvccSnapshot[applied={T|T < 7334145525957238784}]
I20260928 02:34:32.555562 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:34:32.555646 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:34:32.556319 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c7dda90ee7e44575b410d248043a8c0f: Sending DeleteTablet for 3 replicas of tablet 3f6f92b52dfc4252bdec594ec4987490
I20260928 02:34:32.556603 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f6f92b52dfc4252bdec594ec4987490 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49918
I20260928 02:34:32.556639 1783 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f6f92b52dfc4252bdec594ec4987490 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36412
I20260928 02:34:32.556809 1642 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f6f92b52dfc4252bdec594ec4987490 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33162
I20260928 02:34:32.556847 2039 tablet_replica.cc:330] T 3f6f92b52dfc4252bdec594ec4987490 P 03510eb2c310483fa77baf78af35dbef: stopping tablet replica
I20260928 02:34:32.556931 2039 raft_consensus.cc:2229] T 3f6f92b52dfc4252bdec594ec4987490 P 03510eb2c310483fa77baf78af35dbef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:34:32.556989 2038 tablet_replica.cc:330] T 3f6f92b52dfc4252bdec594ec4987490 P cfec3cfacb084f148da0bdc30378ea79: stopping tablet replica
I20260928 02:34:32.557041 2039 raft_consensus.cc:2258] T 3f6f92b52dfc4252bdec594ec4987490 P 03510eb2c310483fa77baf78af35dbef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:32.557058 2038 raft_consensus.cc:2229] T 3f6f92b52dfc4252bdec594ec4987490 P cfec3cfacb084f148da0bdc30378ea79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:34:32.557201 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:34:32.557415 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:34:32.557415 2040 tablet_replica.cc:330] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5: stopping tablet replica
I20260928 02:34:32.557523 2040 raft_consensus.cc:2229] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:34:32.557627 2040 raft_consensus.cc:2258] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:32.557879 2038 raft_consensus.cc:2258] T 3f6f92b52dfc4252bdec594ec4987490 P cfec3cfacb084f148da0bdc30378ea79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:32.558374 2039 ts_tablet_manager.cc:1905] T 3f6f92b52dfc4252bdec594ec4987490 P 03510eb2c310483fa77baf78af35dbef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:32.558702 1642 tablet_service.cc:1467] Processing CreateTablet for tablet aceb9cbdfde04659b7fb89102ed76a9a (DEFAULT_TABLE table=test-table [id=1236328b9abf409f95ee9c33ae7f2ea6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:32.558751 2040 ts_tablet_manager.cc:1905] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:32.558823 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aceb9cbdfde04659b7fb89102ed76a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:32.559233 2038 ts_tablet_manager.cc:1905] T 3f6f92b52dfc4252bdec594ec4987490 P cfec3cfacb084f148da0bdc30378ea79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:32.560043 2039 ts_tablet_manager.cc:1918] T 3f6f92b52dfc4252bdec594ec4987490 P 03510eb2c310483fa77baf78af35dbef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:34:32.560101 2039 log.cc:1196] T 3f6f92b52dfc4252bdec594ec4987490 P 03510eb2c310483fa77baf78af35dbef: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-1/wal/wals/3f6f92b52dfc4252bdec594ec4987490
I20260928 02:34:32.560117 1924 tablet_service.cc:1467] Processing CreateTablet for tablet aceb9cbdfde04659b7fb89102ed76a9a (DEFAULT_TABLE table=test-table [id=1236328b9abf409f95ee9c33ae7f2ea6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:32.560213 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aceb9cbdfde04659b7fb89102ed76a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:32.560316 2039 ts_tablet_manager.cc:1939] T 3f6f92b52dfc4252bdec594ec4987490 P 03510eb2c310483fa77baf78af35dbef: Deleting consensus metadata
I20260928 02:34:32.560663 2038 ts_tablet_manager.cc:1918] T 3f6f92b52dfc4252bdec594ec4987490 P cfec3cfacb084f148da0bdc30378ea79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:34:32.560719 2038 log.cc:1196] T 3f6f92b52dfc4252bdec594ec4987490 P cfec3cfacb084f148da0bdc30378ea79: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-2/wal/wals/3f6f92b52dfc4252bdec594ec4987490
I20260928 02:34:32.560909 2038 ts_tablet_manager.cc:1939] T 3f6f92b52dfc4252bdec594ec4987490 P cfec3cfacb084f148da0bdc30378ea79: Deleting consensus metadata
I20260928 02:34:32.561342 1490 catalog_manager.cc:4841] TS cfec3cfacb084f148da0bdc30378ea79 (127.0.98.3:46255): tablet 3f6f92b52dfc4252bdec594ec4987490 (table test-table [id=0689e201f1564e35827688c4938cfb80]) successfully deleted
I20260928 02:34:32.561425 2015 tablet_bootstrap.cc:492] T aceb9cbdfde04659b7fb89102ed76a9a P cc80b62d7fdb4fa0902c077dd91948d5: Bootstrap starting.
I20260928 02:34:32.561450 2017 tablet_bootstrap.cc:492] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79: Bootstrap starting.
I20260928 02:34:32.561796 2015 tablet_bootstrap.cc:654] T aceb9cbdfde04659b7fb89102ed76a9a P cc80b62d7fdb4fa0902c077dd91948d5: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:32.561841 2017 tablet_bootstrap.cc:654] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:32.562271 1491 catalog_manager.cc:4841] TS 03510eb2c310483fa77baf78af35dbef (127.0.98.2:38343): tablet 3f6f92b52dfc4252bdec594ec4987490 (table test-table [id=0689e201f1564e35827688c4938cfb80]) successfully deleted
I20260928 02:34:32.562331 1783 tablet_service.cc:1467] Processing CreateTablet for tablet aceb9cbdfde04659b7fb89102ed76a9a (DEFAULT_TABLE table=test-table [id=1236328b9abf409f95ee9c33ae7f2ea6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:32.562417 1783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aceb9cbdfde04659b7fb89102ed76a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:32.562546 2040 ts_tablet_manager.cc:1918] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:34:32.562597 2040 log.cc:1196] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-0/wal/wals/3f6f92b52dfc4252bdec594ec4987490
I20260928 02:34:32.562805 2040 ts_tablet_manager.cc:1939] T 3f6f92b52dfc4252bdec594ec4987490 P cc80b62d7fdb4fa0902c077dd91948d5: Deleting consensus metadata
I20260928 02:34:32.563175 1490 catalog_manager.cc:4841] TS cc80b62d7fdb4fa0902c077dd91948d5 (127.0.98.1:33325): tablet 3f6f92b52dfc4252bdec594ec4987490 (table test-table [id=0689e201f1564e35827688c4938cfb80]) successfully deleted
I20260928 02:34:32.563477 2014 tablet_bootstrap.cc:492] T aceb9cbdfde04659b7fb89102ed76a9a P 03510eb2c310483fa77baf78af35dbef: Bootstrap starting.
I20260928 02:34:32.563827 2017 tablet_bootstrap.cc:492] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79: No bootstrap required, opened a new log
I20260928 02:34:32.563844 2014 tablet_bootstrap.cc:654] T aceb9cbdfde04659b7fb89102ed76a9a P 03510eb2c310483fa77baf78af35dbef: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:32.563867 2017 ts_tablet_manager.cc:1397] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:34:32.563995 2017 raft_consensus.cc:348] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } }
I20260928 02:34:32.564057 2017 raft_consensus.cc:374] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:32.564075 2017 raft_consensus.cc:729] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfec3cfacb084f148da0bdc30378ea79, State: Initialized, Role: FOLLOWER
I20260928 02:34:32.564119 2017 consensus_queue.cc:260] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } }
I20260928 02:34:32.564200 2017 ts_tablet_manager.cc:1428] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:32.565618 2015 tablet_bootstrap.cc:492] T aceb9cbdfde04659b7fb89102ed76a9a P cc80b62d7fdb4fa0902c077dd91948d5: No bootstrap required, opened a new log
I20260928 02:34:32.565675 2015 ts_tablet_manager.cc:1397] T aceb9cbdfde04659b7fb89102ed76a9a P cc80b62d7fdb4fa0902c077dd91948d5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 02:34:32.565806 2015 raft_consensus.cc:348] T aceb9cbdfde04659b7fb89102ed76a9a P cc80b62d7fdb4fa0902c077dd91948d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } }
I20260928 02:34:32.565874 2015 raft_consensus.cc:374] T aceb9cbdfde04659b7fb89102ed76a9a P cc80b62d7fdb4fa0902c077dd91948d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:32.565920 2015 raft_consensus.cc:729] T aceb9cbdfde04659b7fb89102ed76a9a P cc80b62d7fdb4fa0902c077dd91948d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc80b62d7fdb4fa0902c077dd91948d5, State: Initialized, Role: FOLLOWER
I20260928 02:34:32.565964 2015 consensus_queue.cc:260] T aceb9cbdfde04659b7fb89102ed76a9a P cc80b62d7fdb4fa0902c077dd91948d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } }
I20260928 02:34:32.566062 2015 ts_tablet_manager.cc:1428] T aceb9cbdfde04659b7fb89102ed76a9a P cc80b62d7fdb4fa0902c077dd91948d5: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:32.566373 2014 tablet_bootstrap.cc:492] T aceb9cbdfde04659b7fb89102ed76a9a P 03510eb2c310483fa77baf78af35dbef: No bootstrap required, opened a new log
I20260928 02:34:32.566421 2014 ts_tablet_manager.cc:1397] T aceb9cbdfde04659b7fb89102ed76a9a P 03510eb2c310483fa77baf78af35dbef: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 02:34:32.566646 2014 raft_consensus.cc:348] T aceb9cbdfde04659b7fb89102ed76a9a P 03510eb2c310483fa77baf78af35dbef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } }
I20260928 02:34:32.566704 2014 raft_consensus.cc:374] T aceb9cbdfde04659b7fb89102ed76a9a P 03510eb2c310483fa77baf78af35dbef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:32.566721 2014 raft_consensus.cc:729] T aceb9cbdfde04659b7fb89102ed76a9a P 03510eb2c310483fa77baf78af35dbef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03510eb2c310483fa77baf78af35dbef, State: Initialized, Role: FOLLOWER
I20260928 02:34:32.566762 2014 consensus_queue.cc:260] T aceb9cbdfde04659b7fb89102ed76a9a P 03510eb2c310483fa77baf78af35dbef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } }
I20260928 02:34:32.566852 2014 ts_tablet_manager.cc:1428] T aceb9cbdfde04659b7fb89102ed76a9a P 03510eb2c310483fa77baf78af35dbef: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:32.575551 2021 raft_consensus.cc:482] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:34:32.575613 2021 raft_consensus.cc:504] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } }
I20260928 02:34:32.575887 2021 leader_election.cc:288] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc80b62d7fdb4fa0902c077dd91948d5 (127.0.98.1:33325), 03510eb2c310483fa77baf78af35dbef (127.0.98.2:38343)
I20260928 02:34:32.579563 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:32.576136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42391 (local address 127.0.98.2:38343)
0928 02:34:32.576180 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:32.576182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:32.576409 (+ 227us) server_negotiation.cc:407] Connection header received
0928 02:34:32.576502 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:32.576503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:32.576512 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:32.576538 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:32.576695 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.577209 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.578702 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.579199 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.579233 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:32.579241 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:32.579445 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 02:34:32.579495 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 02:34:32.579571 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:32.576023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33325 (local address 127.0.98.3:39317)
0928 02:34:32.576078 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:32.576089 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:32.576132 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:32.576372 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:32.576373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:32.576374 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:32.576454 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:32.576456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.577455 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.577456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:32.578632 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:32.578634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.579355 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.579356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:32.579423 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:32.579426 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:34:32.579476 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 02:34:32.579512 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 02:34:32.579596 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:32.576185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39317 (local address 127.0.98.1:33325)
0928 02:34:32.576232 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:32.576233 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:32.576240 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:34:32.576261 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:32.576262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:32.576279 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:32.576324 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:32.576527 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.577348 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.578715 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.579273 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.579352 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:32.579361 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:32.579487 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 02:34:32.579523 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 02:34:32.579826 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aceb9cbdfde04659b7fb89102ed76a9a" candidate_uuid: "cfec3cfacb084f148da0bdc30378ea79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" is_pre_election: true
I20260928 02:34:32.579974 1662 raft_consensus.cc:2454] T aceb9cbdfde04659b7fb89102ed76a9a P cc80b62d7fdb4fa0902c077dd91948d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfec3cfacb084f148da0bdc30378ea79 in term 0.
I20260928 02:34:32.580109 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:32.576032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38343 (local address 127.0.98.3:42391)
0928 02:34:32.576327 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:32.576336 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:32.576385 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:32.576586 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:32.576587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:32.576588 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:32.576642 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:32.576643 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.577294 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.577295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:32.578605 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:32.578608 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.579306 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.579307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:32.579370 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:32.579373 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:34:32.579996 (+ 623us) client_negotiation.cc:240] Negotiation successful
0928 02:34:32.580040 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":226,"thread_start_us":64,"threads_started":1}
I20260928 02:34:32.580144 1876 leader_election.cc:302] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc80b62d7fdb4fa0902c077dd91948d5, cfec3cfacb084f148da0bdc30378ea79; no voters:
I20260928 02:34:32.580264 2021 raft_consensus.cc:2781] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:34:32.580299 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aceb9cbdfde04659b7fb89102ed76a9a" candidate_uuid: "cfec3cfacb084f148da0bdc30378ea79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03510eb2c310483fa77baf78af35dbef" is_pre_election: true
I20260928 02:34:32.580376 2021 raft_consensus.cc:482] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:34:32.580381 1803 raft_consensus.cc:2454] T aceb9cbdfde04659b7fb89102ed76a9a P 03510eb2c310483fa77baf78af35dbef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfec3cfacb084f148da0bdc30378ea79 in term 0.
I20260928 02:34:32.580403 2021 raft_consensus.cc:3037] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:32.581328 2021 raft_consensus.cc:504] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } }
I20260928 02:34:32.581466 2021 leader_election.cc:288] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79 [CANDIDATE]: Term 1 election: Requested vote from peers cc80b62d7fdb4fa0902c077dd91948d5 (127.0.98.1:33325), 03510eb2c310483fa77baf78af35dbef (127.0.98.2:38343)
I20260928 02:34:32.581588 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aceb9cbdfde04659b7fb89102ed76a9a" candidate_uuid: "cfec3cfacb084f148da0bdc30378ea79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03510eb2c310483fa77baf78af35dbef"
I20260928 02:34:32.581676 1803 raft_consensus.cc:3037] T aceb9cbdfde04659b7fb89102ed76a9a P 03510eb2c310483fa77baf78af35dbef [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:32.582270 1803 raft_consensus.cc:2454] T aceb9cbdfde04659b7fb89102ed76a9a P 03510eb2c310483fa77baf78af35dbef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfec3cfacb084f148da0bdc30378ea79 in term 1.
I20260928 02:34:32.582453 1877 leader_election.cc:302] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03510eb2c310483fa77baf78af35dbef, cfec3cfacb084f148da0bdc30378ea79; no voters:
I20260928 02:34:32.582456 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aceb9cbdfde04659b7fb89102ed76a9a" candidate_uuid: "cfec3cfacb084f148da0bdc30378ea79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc80b62d7fdb4fa0902c077dd91948d5"
I20260928 02:34:32.582530 1662 raft_consensus.cc:3037] T aceb9cbdfde04659b7fb89102ed76a9a P cc80b62d7fdb4fa0902c077dd91948d5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:32.582541 2021 raft_consensus.cc:2781] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:32.582652 2021 raft_consensus.cc:686] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79 [term 1 LEADER]: Becoming Leader. State: Replica: cfec3cfacb084f148da0bdc30378ea79, State: Running, Role: LEADER
I20260928 02:34:32.582710 2021 consensus_queue.cc:237] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } }
I20260928 02:34:32.583328 1662 raft_consensus.cc:2454] T aceb9cbdfde04659b7fb89102ed76a9a P cc80b62d7fdb4fa0902c077dd91948d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfec3cfacb084f148da0bdc30378ea79 in term 1.
I20260928 02:34:32.583319 1503 catalog_manager.cc:5495] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79 reported cstate change: term changed from 0 to 1, leader changed from <none> to cfec3cfacb084f148da0bdc30378ea79 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "cfec3cfacb084f148da0bdc30378ea79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } health_report { overall_health: HEALTHY } } }
Sep 28 02:34:32 dist-test-slave-fz47 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:32.621366 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:32.613956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44961 (local address 127.0.0.1:48976)
0928 02:34:32.614115 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:32.614125 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:32.614184 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:32.614327 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:32.614327 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:32.614619 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:32.614663 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:32.614665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.615594 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.615595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:32.616141 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:32.616143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.616633 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.616634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:32.616691 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:32.617204 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:32.617208 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:32.618759 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:32.620369 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:32.620375 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:32.620377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:32.620566 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:32.620761 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:32.620761 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:32.620761 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:32.620810 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:32.621059 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:32.621064 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:32.621102 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:34:32.621147 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 02:34:32.621230 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":92,"threads_started":1}
I20260928 02:34:32.621440 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:32.614030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48976 (local address 127.0.98.62:44961)
0928 02:34:32.614094 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:32.614096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:32.614181 (+ 85us) server_negotiation.cc:407] Connection header received
0928 02:34:32.614229 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:32.614230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:32.614240 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:32.614318 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:32.614727 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.615483 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.616224 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.616487 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.616562 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:32.618847 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:32.618848 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:32.618849 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:32.618856 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:32.620295 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:32.620634 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:32.620636 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:32.620636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:32.620699 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:32.620865 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:32.620865 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:32.620866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:32.620999 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:32.621072 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:32.621190 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 02:34:32.621314 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":33}
I20260928 02:34:32.623293 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48976:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:34:32.623481 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:34:32.624818 1924 tablet_service.cc:1467] Processing CreateTablet for tablet be2a27699bfa4961a8a50dea90ab8632 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=58d063f73e054a478fe30f824c0bb969]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:34:32.624855 1642 tablet_service.cc:1467] Processing CreateTablet for tablet be2a27699bfa4961a8a50dea90ab8632 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=58d063f73e054a478fe30f824c0bb969]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:34:32.624965 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be2a27699bfa4961a8a50dea90ab8632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:32.624984 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be2a27699bfa4961a8a50dea90ab8632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:32.625526 1783 tablet_service.cc:1467] Processing CreateTablet for tablet be2a27699bfa4961a8a50dea90ab8632 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=58d063f73e054a478fe30f824c0bb969]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:34:32.625658 1783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be2a27699bfa4961a8a50dea90ab8632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:32.626576 2015 tablet_bootstrap.cc:492] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5: Bootstrap starting.
I20260928 02:34:32.626650 2017 tablet_bootstrap.cc:492] T be2a27699bfa4961a8a50dea90ab8632 P cfec3cfacb084f148da0bdc30378ea79: Bootstrap starting.
I20260928 02:34:32.627002 2015 tablet_bootstrap.cc:654] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:32.627050 2017 tablet_bootstrap.cc:654] T be2a27699bfa4961a8a50dea90ab8632 P cfec3cfacb084f148da0bdc30378ea79: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:32.627373 2014 tablet_bootstrap.cc:492] T be2a27699bfa4961a8a50dea90ab8632 P 03510eb2c310483fa77baf78af35dbef: Bootstrap starting.
I20260928 02:34:32.627758 2014 tablet_bootstrap.cc:654] T be2a27699bfa4961a8a50dea90ab8632 P 03510eb2c310483fa77baf78af35dbef: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:32.627867 2017 tablet_bootstrap.cc:492] T be2a27699bfa4961a8a50dea90ab8632 P cfec3cfacb084f148da0bdc30378ea79: No bootstrap required, opened a new log
I20260928 02:34:32.627899 2017 ts_tablet_manager.cc:1397] T be2a27699bfa4961a8a50dea90ab8632 P cfec3cfacb084f148da0bdc30378ea79: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:34:32.628230 2017 raft_consensus.cc:348] T be2a27699bfa4961a8a50dea90ab8632 P cfec3cfacb084f148da0bdc30378ea79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } }
I20260928 02:34:32.628309 2017 raft_consensus.cc:374] T be2a27699bfa4961a8a50dea90ab8632 P cfec3cfacb084f148da0bdc30378ea79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:32.628381 2017 raft_consensus.cc:729] T be2a27699bfa4961a8a50dea90ab8632 P cfec3cfacb084f148da0bdc30378ea79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfec3cfacb084f148da0bdc30378ea79, State: Initialized, Role: FOLLOWER
I20260928 02:34:32.628432 2014 tablet_bootstrap.cc:492] T be2a27699bfa4961a8a50dea90ab8632 P 03510eb2c310483fa77baf78af35dbef: No bootstrap required, opened a new log
I20260928 02:34:32.628424 2017 consensus_queue.cc:260] T be2a27699bfa4961a8a50dea90ab8632 P cfec3cfacb084f148da0bdc30378ea79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } }
I20260928 02:34:32.628473 2014 ts_tablet_manager.cc:1397] T be2a27699bfa4961a8a50dea90ab8632 P 03510eb2c310483fa77baf78af35dbef: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 02:34:32.628512 2017 ts_tablet_manager.cc:1428] T be2a27699bfa4961a8a50dea90ab8632 P cfec3cfacb084f148da0bdc30378ea79: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:34:32.628561 2045 tablet_replica.cc:439] T be2a27699bfa4961a8a50dea90ab8632 P cfec3cfacb084f148da0bdc30378ea79: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } } }
I20260928 02:34:32.628628 2045 tablet_replica.cc:442] T be2a27699bfa4961a8a50dea90ab8632 P cfec3cfacb084f148da0bdc30378ea79: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:34:32.628599 2014 raft_consensus.cc:348] T be2a27699bfa4961a8a50dea90ab8632 P 03510eb2c310483fa77baf78af35dbef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } }
I20260928 02:34:32.628662 2014 raft_consensus.cc:374] T be2a27699bfa4961a8a50dea90ab8632 P 03510eb2c310483fa77baf78af35dbef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:32.628680 2014 raft_consensus.cc:729] T be2a27699bfa4961a8a50dea90ab8632 P 03510eb2c310483fa77baf78af35dbef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03510eb2c310483fa77baf78af35dbef, State: Initialized, Role: FOLLOWER
I20260928 02:34:32.628720 2014 consensus_queue.cc:260] T be2a27699bfa4961a8a50dea90ab8632 P 03510eb2c310483fa77baf78af35dbef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } }
I20260928 02:34:32.628813 2014 ts_tablet_manager.cc:1428] T be2a27699bfa4961a8a50dea90ab8632 P 03510eb2c310483fa77baf78af35dbef: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:32.628935 2020 tablet_replica.cc:439] T be2a27699bfa4961a8a50dea90ab8632 P 03510eb2c310483fa77baf78af35dbef: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } } }
I20260928 02:34:32.629000 2020 tablet_replica.cc:442] T be2a27699bfa4961a8a50dea90ab8632 P 03510eb2c310483fa77baf78af35dbef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:34:32.629590 2015 tablet_bootstrap.cc:492] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5: No bootstrap required, opened a new log
I20260928 02:34:32.629647 2015 ts_tablet_manager.cc:1397] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 02:34:32.629771 2015 raft_consensus.cc:348] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } }
I20260928 02:34:32.629856 2015 raft_consensus.cc:374] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:32.629884 2015 raft_consensus.cc:729] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc80b62d7fdb4fa0902c077dd91948d5, State: Initialized, Role: FOLLOWER
I20260928 02:34:32.629933 2015 consensus_queue.cc:260] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } }
I20260928 02:34:32.630023 2015 ts_tablet_manager.cc:1428] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:32.630122 2022 tablet_replica.cc:439] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } } }
I20260928 02:34:32.630191 2022 tablet_replica.cc:442] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 02:34:32.662534 1991 tablet.cc:2369] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:34:32.883864 2022 raft_consensus.cc:482] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:34:32.883965 2022 raft_consensus.cc:504] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } }
I20260928 02:34:32.884123 2022 leader_election.cc:288] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03510eb2c310483fa77baf78af35dbef (127.0.98.2:38343), cfec3cfacb084f148da0bdc30378ea79 (127.0.98.3:46255)
I20260928 02:34:32.884327 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be2a27699bfa4961a8a50dea90ab8632" candidate_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03510eb2c310483fa77baf78af35dbef" is_pre_election: true
I20260928 02:34:32.884335 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be2a27699bfa4961a8a50dea90ab8632" candidate_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfec3cfacb084f148da0bdc30378ea79" is_pre_election: true
I20260928 02:34:32.884419 1803 raft_consensus.cc:2454] T be2a27699bfa4961a8a50dea90ab8632 P 03510eb2c310483fa77baf78af35dbef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc80b62d7fdb4fa0902c077dd91948d5 in term 0.
I20260928 02:34:32.884424 1944 raft_consensus.cc:2454] T be2a27699bfa4961a8a50dea90ab8632 P cfec3cfacb084f148da0bdc30378ea79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc80b62d7fdb4fa0902c077dd91948d5 in term 0.
I20260928 02:34:32.884568 1595 leader_election.cc:302] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03510eb2c310483fa77baf78af35dbef, cc80b62d7fdb4fa0902c077dd91948d5; no voters:
I20260928 02:34:32.884650 2022 raft_consensus.cc:2781] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:34:32.884696 2022 raft_consensus.cc:482] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:34:32.884716 2022 raft_consensus.cc:3037] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:32.885437 2022 raft_consensus.cc:504] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } }
I20260928 02:34:32.885557 2022 leader_election.cc:288] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5 [CANDIDATE]: Term 1 election: Requested vote from peers 03510eb2c310483fa77baf78af35dbef (127.0.98.2:38343), cfec3cfacb084f148da0bdc30378ea79 (127.0.98.3:46255)
I20260928 02:34:32.885702 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be2a27699bfa4961a8a50dea90ab8632" candidate_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03510eb2c310483fa77baf78af35dbef"
I20260928 02:34:32.885775 1803 raft_consensus.cc:3037] T be2a27699bfa4961a8a50dea90ab8632 P 03510eb2c310483fa77baf78af35dbef [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:32.885742 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be2a27699bfa4961a8a50dea90ab8632" candidate_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfec3cfacb084f148da0bdc30378ea79"
I20260928 02:34:32.885810 1944 raft_consensus.cc:3037] T be2a27699bfa4961a8a50dea90ab8632 P cfec3cfacb084f148da0bdc30378ea79 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:32.886399 1803 raft_consensus.cc:2454] T be2a27699bfa4961a8a50dea90ab8632 P 03510eb2c310483fa77baf78af35dbef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc80b62d7fdb4fa0902c077dd91948d5 in term 1.
I20260928 02:34:32.886466 1944 raft_consensus.cc:2454] T be2a27699bfa4961a8a50dea90ab8632 P cfec3cfacb084f148da0bdc30378ea79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc80b62d7fdb4fa0902c077dd91948d5 in term 1.
I20260928 02:34:32.886551 1595 leader_election.cc:302] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03510eb2c310483fa77baf78af35dbef, cc80b62d7fdb4fa0902c077dd91948d5; no voters:
I20260928 02:34:32.886653 2022 raft_consensus.cc:2781] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:32.886704 2022 raft_consensus.cc:686] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5 [term 1 LEADER]: Becoming Leader. State: Replica: cc80b62d7fdb4fa0902c077dd91948d5, State: Running, Role: LEADER
I20260928 02:34:32.886759 2022 consensus_queue.cc:237] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } }
I20260928 02:34:32.887017 2035 tablet_replica.cc:439] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5: TxnStatusTablet state changed. Reason: New leader cc80b62d7fdb4fa0902c077dd91948d5. Latest consensus state: current_term: 1 leader_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } } }
I20260928 02:34:32.887094 2035 tablet_replica.cc:442] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:34:32.887271 2057 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:34:32.887368 2057 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 02:34:32.887310 1503 catalog_manager.cc:5495] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc80b62d7fdb4fa0902c077dd91948d5 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } health_report { overall_health: UNKNOWN } } }
I20260928 02:34:32.915135 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:32.912705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33190 (local address 127.0.98.1:33325)
0928 02:34:32.912763 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:32.912765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:32.912772 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:34:32.912792 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:32.912792 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:32.912803 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:32.912858 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:34:32.912996 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.913505 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.914554 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.914726 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.914770 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:32.914934 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:34:32.914966 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:34:32.914995 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:32.915065 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 02:34:32.915078 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:34:32.915189 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:32.912465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33325 (local address 127.0.0.1:33190)
0928 02:34:32.912539 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:32.912552 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:32.912610 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:32.912882 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:32.912882 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:32.912884 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:34:32.912934 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:32.912937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.913567 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.913567 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:32.914483 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:32.914486 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.914783 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.914783 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:32.914848 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:32.914865 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:34:32.915010 (+ 145us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:34:32.915015 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:34:32.915062 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 02:34:32.915110 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 02:34:32.916127 1944 raft_consensus.cc:1261] T be2a27699bfa4961a8a50dea90ab8632 P cfec3cfacb084f148da0bdc30378ea79 [term 1 FOLLOWER]: Refusing update from remote peer cc80b62d7fdb4fa0902c077dd91948d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:32.916148 1803 raft_consensus.cc:1261] T be2a27699bfa4961a8a50dea90ab8632 P 03510eb2c310483fa77baf78af35dbef [term 1 FOLLOWER]: Refusing update from remote peer cc80b62d7fdb4fa0902c077dd91948d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:32.916347 2022 consensus_queue.cc:1035] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:32.916445 2035 consensus_queue.cc:1035] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:32.916909 2045 tablet_replica.cc:439] T be2a27699bfa4961a8a50dea90ab8632 P cfec3cfacb084f148da0bdc30378ea79: TxnStatusTablet state changed. Reason: New leader cc80b62d7fdb4fa0902c077dd91948d5. Latest consensus state: current_term: 1 leader_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } } }
I20260928 02:34:32.916991 2045 tablet_replica.cc:442] T be2a27699bfa4961a8a50dea90ab8632 P cfec3cfacb084f148da0bdc30378ea79: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:34:32.917008 2020 tablet_replica.cc:439] T be2a27699bfa4961a8a50dea90ab8632 P 03510eb2c310483fa77baf78af35dbef: TxnStatusTablet state changed. Reason: New leader cc80b62d7fdb4fa0902c077dd91948d5. Latest consensus state: current_term: 1 leader_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } } }
I20260928 02:34:32.917095 2020 tablet_replica.cc:442] T be2a27699bfa4961a8a50dea90ab8632 P 03510eb2c310483fa77baf78af35dbef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:34:32.917559 2035 tablet_replica.cc:439] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } } }
I20260928 02:34:32.917629 2035 tablet_replica.cc:442] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:34:32.917680 2028 mvcc.cc:205] Tried to move back new op lower bound from 7334145527463149568 to 7334145527344603136. Current Snapshot: MvccSnapshot[applied={T|T < 7334145527463149568 or (T in {7334145527463149568})}]
I20260928 02:34:32.917999 2045 tablet_replica.cc:439] T be2a27699bfa4961a8a50dea90ab8632 P cfec3cfacb084f148da0bdc30378ea79: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } } }
I20260928 02:34:32.918073 2045 tablet_replica.cc:442] T be2a27699bfa4961a8a50dea90ab8632 P cfec3cfacb084f148da0bdc30378ea79: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:34:32.918390 2035 tablet_replica.cc:439] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } } }
I20260928 02:34:32.918474 2035 tablet_replica.cc:442] T be2a27699bfa4961a8a50dea90ab8632 P cc80b62d7fdb4fa0902c077dd91948d5: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:34:32.918947 2020 tablet_replica.cc:439] T be2a27699bfa4961a8a50dea90ab8632 P 03510eb2c310483fa77baf78af35dbef: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 } } peers { permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 } } peers { permanent_uuid: "cfec3cfacb084f148da0bdc30378ea79" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46255 } } }
I20260928 02:34:32.919032 2020 tablet_replica.cc:442] T be2a27699bfa4961a8a50dea90ab8632 P 03510eb2c310483fa77baf78af35dbef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:34:32.918965 2030 mvcc.cc:205] Tried to move back new op lower bound from 7334145527463149568 to 7334145527344603136. Current Snapshot: MvccSnapshot[applied={T|T < 7334145527463149568}]
I20260928 02:34:32.922000 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:32.919661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49928 (local address 127.0.98.3:46255)
0928 02:34:32.919693 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:32.919694 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:32.919701 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:34:32.919754 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:32.919755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:32.919771 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:32.919838 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:34:32.919930 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.920369 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.921384 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.921565 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.921613 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:32.921749 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:34:32.921836 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:34:32.921879 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:32.921931 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 02:34:32.921952 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 02:34:32.926553 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:32.924103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35241 (local address 127.0.98.1:33325)
0928 02:34:32.924147 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:32.924149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:32.924194 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:34:32.924254 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:32.924254 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:32.924263 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:32.924305 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:34:32.924424 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.924865 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.925947 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.926123 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.926172 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:32.926374 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:34:32.926403 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:34:32.926443 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:32.926488 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 02:34:32.926502 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 02:34:32.926626 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:32.923978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33325 (local address 127.0.98.3:35241)
0928 02:34:32.924121 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:32.924132 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:32.924208 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:32.924309 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:32.924309 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:32.924311 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:34:32.924370 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:32.924372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.924927 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.924928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:32.925881 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:32.925883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.926200 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.926200 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:32.926268 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:32.926289 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:34:32.926441 (+ 152us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:34:32.926442 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 02:34:32.926486 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 02:34:32.926533 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
I20260928 02:34:32.928517 1803 raft_consensus.cc:1261] T aceb9cbdfde04659b7fb89102ed76a9a P 03510eb2c310483fa77baf78af35dbef [term 1 FOLLOWER]: Refusing update from remote peer cfec3cfacb084f148da0bdc30378ea79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:32.928710 1662 raft_consensus.cc:1261] T aceb9cbdfde04659b7fb89102ed76a9a P cc80b62d7fdb4fa0902c077dd91948d5 [term 1 FOLLOWER]: Refusing update from remote peer cfec3cfacb084f148da0bdc30378ea79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:32.928740 2045 consensus_queue.cc:1035] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03510eb2c310483fa77baf78af35dbef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:32.928942 2045 consensus_queue.cc:1035] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc80b62d7fdb4fa0902c077dd91948d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:32.940497 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:32.937634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60761 (local address 127.0.98.3:46255)
0928 02:34:32.937678 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:32.937679 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:32.937808 (+ 129us) server_negotiation.cc:407] Connection header received
0928 02:34:32.937850 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:32.937851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:32.937859 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:32.937891 (+ 32us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:34:32.938050 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.938776 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.939834 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.940017 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.940068 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:32.940248 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:34:32.940282 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:34:32.940347 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:32.940404 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 02:34:32.940428 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 02:34:32.940523 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:32.937588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46255 (local address 127.0.98.1:60761)
0928 02:34:32.937739 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:32.937749 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:32.937803 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:32.937933 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:32.937934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:32.937935 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:34:32.937978 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:32.937980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.938868 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.938869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:32.939762 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:32.939765 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.940081 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.940081 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:32.940137 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:32.940152 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:34:32.940354 (+ 202us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:34:32.940356 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:34:32.940397 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 02:34:32.940444 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":68,"threads_started":1}
I20260928 02:34:32.949805 1504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:34:32.949894 1504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:34:32.950404 1504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c7dda90ee7e44575b410d248043a8c0f: Sending DeleteTablet for 3 replicas of tablet aceb9cbdfde04659b7fb89102ed76a9a
I20260928 02:34:32.950659 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet aceb9cbdfde04659b7fb89102ed76a9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49918
I20260928 02:34:32.950655 1783 tablet_service.cc:1514] Processing DeleteTablet for tablet aceb9cbdfde04659b7fb89102ed76a9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36412
I20260928 02:34:32.950742 2038 tablet_replica.cc:330] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79: stopping tablet replica
I20260928 02:34:32.950790 2038 raft_consensus.cc:2229] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:34:32.950855 2038 raft_consensus.cc:2258] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:32.950928 2039 tablet_replica.cc:330] T aceb9cbdfde04659b7fb89102ed76a9a P 03510eb2c310483fa77baf78af35dbef: stopping tablet replica
I20260928 02:34:32.950989 2039 raft_consensus.cc:2229] T aceb9cbdfde04659b7fb89102ed76a9a P 03510eb2c310483fa77baf78af35dbef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:34:32.951020 2039 raft_consensus.cc:2258] T aceb9cbdfde04659b7fb89102ed76a9a P 03510eb2c310483fa77baf78af35dbef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:32.951078 1642 tablet_service.cc:1514] Processing DeleteTablet for tablet aceb9cbdfde04659b7fb89102ed76a9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33162
I20260928 02:34:32.951150 2040 tablet_replica.cc:330] T aceb9cbdfde04659b7fb89102ed76a9a P cc80b62d7fdb4fa0902c077dd91948d5: stopping tablet replica
I20260928 02:34:32.951196 2040 raft_consensus.cc:2229] T aceb9cbdfde04659b7fb89102ed76a9a P cc80b62d7fdb4fa0902c077dd91948d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:34:32.951224 2040 raft_consensus.cc:2258] T aceb9cbdfde04659b7fb89102ed76a9a P cc80b62d7fdb4fa0902c077dd91948d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:32.951414 2039 ts_tablet_manager.cc:1905] T aceb9cbdfde04659b7fb89102ed76a9a P 03510eb2c310483fa77baf78af35dbef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:32.951445 2040 ts_tablet_manager.cc:1905] T aceb9cbdfde04659b7fb89102ed76a9a P cc80b62d7fdb4fa0902c077dd91948d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:32.953130 2040 ts_tablet_manager.cc:1918] T aceb9cbdfde04659b7fb89102ed76a9a P cc80b62d7fdb4fa0902c077dd91948d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:34:32.953178 2040 log.cc:1196] T aceb9cbdfde04659b7fb89102ed76a9a P cc80b62d7fdb4fa0902c077dd91948d5: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-0/wal/wals/aceb9cbdfde04659b7fb89102ed76a9a
I20260928 02:34:32.953183 2038 ts_tablet_manager.cc:1905] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:32.953382 2040 ts_tablet_manager.cc:1939] T aceb9cbdfde04659b7fb89102ed76a9a P cc80b62d7fdb4fa0902c077dd91948d5: Deleting consensus metadata
I20260928 02:34:32.953689 1490 catalog_manager.cc:4841] TS cc80b62d7fdb4fa0902c077dd91948d5 (127.0.98.1:33325): tablet aceb9cbdfde04659b7fb89102ed76a9a (table test-table [id=1236328b9abf409f95ee9c33ae7f2ea6]) successfully deleted
I20260928 02:34:32.954880 2038 ts_tablet_manager.cc:1918] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:34:32.954916 2038 log.cc:1196] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-2/wal/wals/aceb9cbdfde04659b7fb89102ed76a9a
I20260928 02:34:32.955045 2038 ts_tablet_manager.cc:1939] T aceb9cbdfde04659b7fb89102ed76a9a P cfec3cfacb084f148da0bdc30378ea79: Deleting consensus metadata
I20260928 02:34:32.955246 2039 ts_tablet_manager.cc:1918] T aceb9cbdfde04659b7fb89102ed76a9a P 03510eb2c310483fa77baf78af35dbef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:34:32.955293 2039 log.cc:1196] T aceb9cbdfde04659b7fb89102ed76a9a P 03510eb2c310483fa77baf78af35dbef: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790562862804661-392-0/minicluster-data/ts-1/wal/wals/aceb9cbdfde04659b7fb89102ed76a9a
I20260928 02:34:32.955472 2039 ts_tablet_manager.cc:1939] T aceb9cbdfde04659b7fb89102ed76a9a P 03510eb2c310483fa77baf78af35dbef: Deleting consensus metadata
I20260928 02:34:32.955716 1491 catalog_manager.cc:4841] TS 03510eb2c310483fa77baf78af35dbef (127.0.98.2:38343): tablet aceb9cbdfde04659b7fb89102ed76a9a (table test-table [id=1236328b9abf409f95ee9c33ae7f2ea6]) successfully deleted
I20260928 02:34:32.955857 1490 catalog_manager.cc:4841] TS cfec3cfacb084f148da0bdc30378ea79 (127.0.98.3:46255): tablet aceb9cbdfde04659b7fb89102ed76a9a (table test-table [id=1236328b9abf409f95ee9c33ae7f2ea6]) successfully deleted
Sep 28 02:34:32 dist-test-slave-fz47 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562872, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 02:34:32.964613 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:32.957490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33192 (local address 127.0.98.1:33325)
0928 02:34:32.957531 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:32.957533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:32.957643 (+ 110us) server_negotiation.cc:407] Connection header received
0928 02:34:32.957726 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:32.957728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:32.957737 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:32.957780 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:32.958288 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.958819 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.959494 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.959689 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.959738 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:32.962208 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:32.962216 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:32.962216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:32.962223 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:32.963610 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:32.963904 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:32.963907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:32.963907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:32.963964 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:32.964133 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:32.964134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:32.964134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:32.964259 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:32.964391 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:32.964453 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 02:34:32.964527 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 02:34:32.964816 1702 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33192
I20260928 02:34:32.975347 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:32.969469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48992 (local address 127.0.98.62:44961)
0928 02:34:32.969519 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:32.969521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:32.969551 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:34:32.969606 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:32.969607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:32.969617 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:32.969663 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:32.970211 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.970948 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.971838 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:32.972036 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:32.972092 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:32.973163 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:32.973164 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:32.973164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:32.973171 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:32.974399 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:32.974697 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:32.974698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:32.974698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:32.974751 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:32.974903 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:32.974904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:32.974904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:32.975014 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:32.975098 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:32.975188 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 02:34:32.975259 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 02:34:32.975637 1504 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48992
I20260928 02:34:32.976270 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 1576
I20260928 02:34:32.982554 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 1717
I20260928 02:34:32.987111 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 1858
I20260928 02:34:32.991161 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 1472
2026-09-28T02:34:32Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (6736 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:34:33 dist-test-slave-fz47 krb5kdc[2078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:34:33 dist-test-slave-fz47 krb5kdc[2078](info): set up 2 sockets
Sep 28 02:34:33 dist-test-slave-fz47 krb5kdc[2078](info): commencing operation
krb5kdc: starting...
W20260928 02:34:33.932916 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:34:34 dist-test-slave-fz47 krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:34:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:34:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 02:34:34.040041 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44123
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:34.120388 2091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:34.120609 2091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:34.120639 2091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:34.121817 2091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:34:34.121860 2091 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:34:34.121879 2091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:34.121897 2091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:34.121912 2091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:34:34.121927 2091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:34:34.123421 2091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44123
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:34.123674 2091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:34.123885 2091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:34:34.128660 2091 server_base.cc:1034] running on GCE node
W20260928 02:34:34.128604 2097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:34.128605 2099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:34.128605 2096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:34.129093 2091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:34.129432 2091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:34.130586 2091 hybrid_clock.cc:648] HybridClock initialized: now 1790562874130572 us; error 30 us; skew 500 ppm
Sep 28 02:34:34 dist-test-slave-fz47 krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:34.133081 2091 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:34:34.133548 2091 webserver.cc:460] Webserver started at http://127.0.98.62:40029/ using document root <none> and password file <none>
I20260928 02:34:34.133728 2091 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:34.133767 2091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:34.133863 2091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:34.134951 2091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "3d1911e566f74fc9b0af803869790bcc"
format_stamp: "Formatted at 2026-09-28 02:34:34 on dist-test-slave-fz47"
server_key: "913add1c7d5c141455ce4002aecef9db"
server_key_iv: "299579fab9889f192a731862246ee5d1"
server_key_version: "encryptionkey@0"
I20260928 02:34:34.135241 2091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "3d1911e566f74fc9b0af803869790bcc"
format_stamp: "Formatted at 2026-09-28 02:34:34 on dist-test-slave-fz47"
server_key: "913add1c7d5c141455ce4002aecef9db"
server_key_iv: "299579fab9889f192a731862246ee5d1"
server_key_version: "encryptionkey@0"
I20260928 02:34:34.136445 2091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:34:34.137110 2106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:34.137324 2091 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:34:34.137409 2091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "3d1911e566f74fc9b0af803869790bcc"
format_stamp: "Formatted at 2026-09-28 02:34:34 on dist-test-slave-fz47"
server_key: "913add1c7d5c141455ce4002aecef9db"
server_key_iv: "299579fab9889f192a731862246ee5d1"
server_key_version: "encryptionkey@0"
I20260928 02:34:34.137475 2091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:34.150449 2091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:34.152398 2091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:34.152511 2091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:34.156883 2091 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44123
I20260928 02:34:34.156951 2168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44123 every 8 connection(s)
I20260928 02:34:34.157440 2091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:34:34.158084 2169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:34.160372 2169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d1911e566f74fc9b0af803869790bcc: Bootstrap starting.
I20260928 02:34:34.160966 2169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d1911e566f74fc9b0af803869790bcc: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:34.161228 2169 log.cc:824] T 00000000000000000000000000000000 P 3d1911e566f74fc9b0af803869790bcc: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:34.161892 2169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d1911e566f74fc9b0af803869790bcc: No bootstrap required, opened a new log
I20260928 02:34:34.163937 2169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d1911e566f74fc9b0af803869790bcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d1911e566f74fc9b0af803869790bcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44123 } }
I20260928 02:34:34.164083 2169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d1911e566f74fc9b0af803869790bcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:34.164113 2169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d1911e566f74fc9b0af803869790bcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1911e566f74fc9b0af803869790bcc, State: Initialized, Role: FOLLOWER
I20260928 02:34:34.164249 2169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d1911e566f74fc9b0af803869790bcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d1911e566f74fc9b0af803869790bcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44123 } }
I20260928 02:34:34.164327 2169 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d1911e566f74fc9b0af803869790bcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:34:34.164358 2169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d1911e566f74fc9b0af803869790bcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:34:34.164407 2169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d1911e566f74fc9b0af803869790bcc [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:34.164831 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 2091
I20260928 02:34:34.164906 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:34:34.164983 2169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d1911e566f74fc9b0af803869790bcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d1911e566f74fc9b0af803869790bcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44123 } }
I20260928 02:34:34.165040 392 external_mini_cluster.cc:1428] Setting key bb10f73657763e3e7fe46a2884e4d3f1
I20260928 02:34:34.165081 2169 leader_election.cc:302] T 00000000000000000000000000000000 P 3d1911e566f74fc9b0af803869790bcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d1911e566f74fc9b0af803869790bcc; no voters:
I20260928 02:34:34.165279 2169 leader_election.cc:288] T 00000000000000000000000000000000 P 3d1911e566f74fc9b0af803869790bcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:34:34.165318 2172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d1911e566f74fc9b0af803869790bcc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:34.165479 2172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d1911e566f74fc9b0af803869790bcc [term 1 LEADER]: Becoming Leader. State: Replica: 3d1911e566f74fc9b0af803869790bcc, State: Running, Role: LEADER
I20260928 02:34:34.165550 2169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d1911e566f74fc9b0af803869790bcc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:34:34.165606 2172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d1911e566f74fc9b0af803869790bcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d1911e566f74fc9b0af803869790bcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44123 } }
I20260928 02:34:34.166505 2173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d1911e566f74fc9b0af803869790bcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d1911e566f74fc9b0af803869790bcc. Latest consensus state: current_term: 1 leader_uuid: "3d1911e566f74fc9b0af803869790bcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d1911e566f74fc9b0af803869790bcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44123 } } }
I20260928 02:34:34.166669 2173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d1911e566f74fc9b0af803869790bcc [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:34.166751 2174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d1911e566f74fc9b0af803869790bcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d1911e566f74fc9b0af803869790bcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d1911e566f74fc9b0af803869790bcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44123 } } }
I20260928 02:34:34.166996 2174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d1911e566f74fc9b0af803869790bcc [sys.catalog]: This master's current role is: LEADER
W20260928 02:34:34.167420 2189 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3d1911e566f74fc9b0af803869790bcc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:34:34.167483 2189 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:34:34.167531 2190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:34:34.167665 2190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:34:34.169163 2190 catalog_manager.cc:1347] Generated new cluster ID: 72269f603e3741f681931d3e61907223
I20260928 02:34:34.169217 2190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:34:34 dist-test-slave-fz47 krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:34.175179 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:34.165670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48540 (local address 127.0.98.62:44123)
0928 02:34:34.165901 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:34.165934 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:34.165970 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:34:34.166117 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:34.166125 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:34.166213 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:34.166367 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:34.167144 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.168145 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.168960 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.169498 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.169594 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:34.171694 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:34.171698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:34.171698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:34.171711 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:34.173757 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:34.174070 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:34.174074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:34.174074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:34.174172 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:34.174343 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:34.174344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:34.174344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:34.174595 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:34.174741 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:34.174839 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 02:34:34.174973 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":184,"thread_start_us":69,"threads_started":1}
I20260928 02:34:34.183645 2190 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:34:34.184412 2190 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:34:34.194490 2190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d1911e566f74fc9b0af803869790bcc: Generated new TSK 0
I20260928 02:34:34.194690 2190 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 02:34:34.220233 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44123
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:34.322229 2195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:34.322435 2195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:34.322460 2195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:34.323539 2195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:34.323572 2195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:34.323590 2195 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:34:34.323601 2195 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:34:34.323619 2195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:34:34.325004 2195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:34.325223 2195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:34.325436 2195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:34.329879 2203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:34.329988 2201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:34.330060 2200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:34.330098 2195 server_base.cc:1034] running on GCE node
I20260928 02:34:34.330277 2195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:34.330513 2195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:34.331662 2195 hybrid_clock.cc:648] HybridClock initialized: now 1790562874331644 us; error 28 us; skew 500 ppm
Sep 28 02:34:34 dist-test-slave-fz47 krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:34.334014 2195 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 02:34:34.334410 2195 webserver.cc:460] Webserver started at http://127.0.98.1:44807/ using document root <none> and password file <none>
I20260928 02:34:34.334596 2195 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:34.334636 2195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:34.334748 2195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:34.335707 2195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-0/data/instance:
uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a"
format_stamp: "Formatted at 2026-09-28 02:34:34 on dist-test-slave-fz47"
server_key: "47f1ae55504ca2d3321e1c4b8769ea5d"
server_key_iv: "98c2f18555bc95a2508e395d12c8472e"
server_key_version: "encryptionkey@0"
I20260928 02:34:34.335988 2195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a"
format_stamp: "Formatted at 2026-09-28 02:34:34 on dist-test-slave-fz47"
server_key: "47f1ae55504ca2d3321e1c4b8769ea5d"
server_key_iv: "98c2f18555bc95a2508e395d12c8472e"
server_key_version: "encryptionkey@0"
I20260928 02:34:34.337393 2195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:34:34.338105 2210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:34.338246 2195 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:34.338351 2195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a"
format_stamp: "Formatted at 2026-09-28 02:34:34 on dist-test-slave-fz47"
server_key: "47f1ae55504ca2d3321e1c4b8769ea5d"
server_key_iv: "98c2f18555bc95a2508e395d12c8472e"
server_key_version: "encryptionkey@0"
I20260928 02:34:34.338405 2195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:34.354537 2195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:34.356659 2195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:34.356765 2195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:34.357196 2195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:34.357236 2195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:34.357312 2195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:34.357339 2195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:34.363343 2195 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42235
I20260928 02:34:34.363411 2325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42235 every 8 connection(s)
Sep 28 02:34:34 dist-test-slave-fz47 krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:34.363716 2195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
I20260928 02:34:34.365118 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 2195
I20260928 02:34:34.365216 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-0/wal/instance
I20260928 02:34:34.365432 392 external_mini_cluster.cc:1428] Setting key 6ddb847f7a6688f918343661ad43c077
I20260928 02:34:34.366566 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:34.357646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42155 (local address 127.0.98.62:44123)
0928 02:34:34.357708 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:34.357710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:34.357906 (+ 196us) server_negotiation.cc:407] Connection header received
0928 02:34:34.358399 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:34.358400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:34.358412 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:34.358467 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:34.359391 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.359930 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.361354 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.361660 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.361720 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:34.363981 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:34.363983 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:34.363983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:34.363994 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:34.365113 (+ 1119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:34.365598 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:34.365599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:34.365600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:34.365665 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:34.365919 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:34.365919 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:34.365920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:34.366036 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:34.366094 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:34.366430 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 02:34:34.366504 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 02:34:34.366775 2219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:34.357624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.1:42155)
0928 02:34:34.357780 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:34.357819 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:34.358309 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:34.358541 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:34.358545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:34.358886 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:34.359294 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:34.359302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.360016 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.360017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:34.361242 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:34.361244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.361734 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.361735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:34.361869 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:34.362353 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:34.362364 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:34.363903 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:34.365188 (+ 1285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:34.365196 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:34.365198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:34.365522 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:34.365726 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:34.365726 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:34.365727 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:34.365862 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:34.366096 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:34.366102 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:34.366300 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 02:34:34.366409 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 02:34:34.366565 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260928 02:34:34.367255 2326 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44123
I20260928 02:34:34.367430 2326 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:34.367645 2326 heartbeater.cc:511] Master 127.0.98.62:44123 requested a full tablet report, sending...
I20260928 02:34:34.368192 2122 ts_manager.cc:194] Registered new tserver with Master: 2cbf48bcdf314ef8b8c6d89fb9e2150a (127.0.98.1:42235)
I20260928 02:34:34.369138 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42155
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 02:34:34.374068 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:34.368244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47567 (local address 127.0.98.62:44123)
0928 02:34:34.368288 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:34.368290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:34.368538 (+ 248us) server_negotiation.cc:407] Connection header received
0928 02:34:34.368564 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:34.368565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:34.368577 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:34.368643 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:34.369010 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.369749 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.370725 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.370904 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.370960 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:34.371872 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:34.371874 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:34.371874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:34.371879 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:34.373111 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:34.373471 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:34.373472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:34.373472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:34.373516 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:34.373676 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:34.373677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:34.373677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:34.373758 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:34.373813 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:34.373952 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 02:34:34.374010 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:34:34.374130 2333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:34.368192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.1:47567)
0928 02:34:34.368422 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:34.368434 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:34.368496 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:34.368686 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:34.368688 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:34.368879 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:34.368936 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:34.368937 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.369856 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.369857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:34.370639 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:34.370641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.370975 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.370976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:34.371048 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:34.371406 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:34.371412 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:34.371777 (+ 365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:34.373190 (+ 1413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:34.373191 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:34.373192 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:34.373407 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:34.373567 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:34.373567 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:34.373568 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:34.373627 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:34.373815 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:34.373816 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:34.373859 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 02:34:34.373934 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 02:34:34.374024 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":70,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 02:34:34.389379 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44123
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:34.467613 2335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:34.467815 2335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:34.467849 2335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:34.469002 2335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:34.469043 2335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:34.469070 2335 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:34:34.469085 2335 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:34:34.469108 2335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:34:34.470535 2335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:34.470757 2335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:34.470968 2335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:34.475579 2341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:34.475581 2340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:34.475684 2343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:34.475713 2335 server_base.cc:1034] running on GCE node
I20260928 02:34:34.475970 2335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:34.476186 2335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:34.477339 2335 hybrid_clock.cc:648] HybridClock initialized: now 1790562874477324 us; error 32 us; skew 500 ppm
Sep 28 02:34:34 dist-test-slave-fz47 krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:34.479553 2335 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 02:34:34.479952 2335 webserver.cc:460] Webserver started at http://127.0.98.2:43083/ using document root <none> and password file <none>
I20260928 02:34:34.480154 2335 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:34.480201 2335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:34.480347 2335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:34.481362 2335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-1/data/instance:
uuid: "249e2c930f94405f9f41c1e861b60c53"
format_stamp: "Formatted at 2026-09-28 02:34:34 on dist-test-slave-fz47"
server_key: "f9e77b7e7859cffa9b1e0e92152958d7"
server_key_iv: "ed66d689df1226f5db5ded70d1cb464c"
server_key_version: "encryptionkey@0"
I20260928 02:34:34.481678 2335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-1/wal/instance:
uuid: "249e2c930f94405f9f41c1e861b60c53"
format_stamp: "Formatted at 2026-09-28 02:34:34 on dist-test-slave-fz47"
server_key: "f9e77b7e7859cffa9b1e0e92152958d7"
server_key_iv: "ed66d689df1226f5db5ded70d1cb464c"
server_key_version: "encryptionkey@0"
I20260928 02:34:34.482816 2335 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:34:34.483429 2350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:34.483565 2335 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:34.483634 2335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "249e2c930f94405f9f41c1e861b60c53"
format_stamp: "Formatted at 2026-09-28 02:34:34 on dist-test-slave-fz47"
server_key: "f9e77b7e7859cffa9b1e0e92152958d7"
server_key_iv: "ed66d689df1226f5db5ded70d1cb464c"
server_key_version: "encryptionkey@0"
I20260928 02:34:34.483690 2335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:34.495323 2335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:34.496927 2335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:34.497047 2335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:34.497606 2335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:34.497648 2335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:34.497675 2335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:34.497700 2335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:34:34 dist-test-slave-fz47 krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:34.504148 2335 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40965
I20260928 02:34:34.504211 2465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40965 every 8 connection(s)
I20260928 02:34:34.504539 2335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
I20260928 02:34:34.506559 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:34.497903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59401 (local address 127.0.98.62:44123)
0928 02:34:34.497956 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:34.497958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:34.498193 (+ 235us) server_negotiation.cc:407] Connection header received
0928 02:34:34.498654 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:34.498656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:34.498668 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:34.498750 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:34.499350 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.499850 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.500810 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.501067 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.501122 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:34.503925 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:34.503926 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:34.503926 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:34.503933 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:34.505399 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:34.505732 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:34.505733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:34.505734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:34.505789 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:34.505994 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:34.505994 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:34.505995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:34.506102 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:34.506154 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:34.506399 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 02:34:34.506474 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 02:34:34.506695 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:34.497893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.2:59401)
0928 02:34:34.498068 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:34.498105 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:34.498564 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:34.498809 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:34.498814 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:34.499079 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:34.499261 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:34.499265 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.499928 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.499930 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:34.500728 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:34.500731 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.501135 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.501136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:34.501240 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:34.501743 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:34.501754 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:34.503824 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:34.505470 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:34.505476 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:34.505478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:34.505670 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:34.505848 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:34.505849 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:34.505849 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:34.505938 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:34.506154 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:34.506160 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:34.506279 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 02:34:34.506361 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 02:34:34.506502 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":97,"threads_started":1}
I20260928 02:34:34.507108 2466 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44123
I20260928 02:34:34.507189 2466 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:34.507351 2466 heartbeater.cc:511] Master 127.0.98.62:44123 requested a full tablet report, sending...
I20260928 02:34:34.507696 2123 ts_manager.cc:194] Registered new tserver with Master: 249e2c930f94405f9f41c1e861b60c53 (127.0.98.2:40965)
I20260928 02:34:34.508141 2123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59401
I20260928 02:34:34.512405 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:34.508041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55187 (local address 127.0.98.62:44123)
0928 02:34:34.508078 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:34.508080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:34.508142 (+ 62us) server_negotiation.cc:407] Connection header received
0928 02:34:34.508187 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:34.508188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:34.508197 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:34.508248 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:34.508579 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.509111 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.509796 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.509965 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.510010 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:34.510689 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:34.510690 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:34.510690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:34.510696 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:34.511645 (+ 949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:34.511904 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:34.511904 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:34.511904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:34.511944 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:34.512048 (+ 104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:34.512048 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:34.512048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:34.512155 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:34.512213 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:34.512298 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 02:34:34.512350 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:34:34.512415 2472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:34.507975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.2:55187)
0928 02:34:34.508091 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:34.508101 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:34.508152 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:34.508289 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:34.508290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:34.508467 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:34.508512 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:34.508513 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.509181 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.509182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:34.509729 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:34.509731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.510014 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.510014 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:34.510062 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:34.510361 (+ 299us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:34.510365 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:34.510627 (+ 262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:34.511704 (+ 1077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:34.511705 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:34.511705 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:34.511853 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:34.511978 (+ 125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:34.511978 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:34.511978 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:34.512018 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:34.512211 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:34.512211 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:34.512239 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 02:34:34.512283 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 02:34:34.512349 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":73,"threads_started":1}
I20260928 02:34:34.513460 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 2335
I20260928 02:34:34.513556 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-1/wal/instance
I20260928 02:34:34.513768 392 external_mini_cluster.cc:1428] Setting key d3cd51545273e5d0b13424b83f0372fd
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 02:34:34.534173 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44123
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:34.612843 2475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:34.613032 2475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:34.613065 2475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:34.614686 2475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:34.614742 2475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:34.614778 2475 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:34:34.614801 2475 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:34:34.614830 2475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:34:34.616317 2475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:34.616492 2475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:34.616648 2475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:34.621192 2483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:34.621227 2480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:34.621248 2481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:34.621297 2475 server_base.cc:1034] running on GCE node
I20260928 02:34:34.621598 2475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:34.621821 2475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:34.622964 2475 hybrid_clock.cc:648] HybridClock initialized: now 1790562874622951 us; error 27 us; skew 500 ppm
Sep 28 02:34:34 dist-test-slave-fz47 krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:34.625233 2475 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 02:34:34.625643 2475 webserver.cc:460] Webserver started at http://127.0.98.3:33039/ using document root <none> and password file <none>
I20260928 02:34:34.625844 2475 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:34.625890 2475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:34.626003 2475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:34.626948 2475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-2/data/instance:
uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a"
format_stamp: "Formatted at 2026-09-28 02:34:34 on dist-test-slave-fz47"
server_key: "1920308a9704ef5d69bed3d2140c485d"
server_key_iv: "8fa4cfb423986167e9ddd88751154087"
server_key_version: "encryptionkey@0"
I20260928 02:34:34.627262 2475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a"
format_stamp: "Formatted at 2026-09-28 02:34:34 on dist-test-slave-fz47"
server_key: "1920308a9704ef5d69bed3d2140c485d"
server_key_iv: "8fa4cfb423986167e9ddd88751154087"
server_key_version: "encryptionkey@0"
I20260928 02:34:34.628432 2475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:34:34.629007 2490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:34.629145 2475 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:34.629279 2475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a"
format_stamp: "Formatted at 2026-09-28 02:34:34 on dist-test-slave-fz47"
server_key: "1920308a9704ef5d69bed3d2140c485d"
server_key_iv: "8fa4cfb423986167e9ddd88751154087"
server_key_version: "encryptionkey@0"
I20260928 02:34:34.629345 2475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:34.645628 2475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:34.647274 2475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:34.647397 2475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:34.647846 2475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:34.647886 2475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:34.647914 2475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:34.647939 2475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:34:34 dist-test-slave-fz47 krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:34.654359 2475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36225
I20260928 02:34:34.654425 2605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36225 every 8 connection(s)
I20260928 02:34:34.654692 2475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
I20260928 02:34:34.656569 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:34.648265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58137 (local address 127.0.98.62:44123)
0928 02:34:34.648304 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:34.648306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:34.648525 (+ 219us) server_negotiation.cc:407] Connection header received
0928 02:34:34.649000 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:34.649001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:34.649014 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:34.649064 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:34.649686 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.650164 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.651145 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.651329 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.651383 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:34.654286 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:34.654288 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:34.654288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:34.654296 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:34.655591 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:34.655932 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:34.655933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:34.655934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:34.655977 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:34.656127 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:34.656128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:34.656128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:34.656218 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:34.656261 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:34.656456 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 02:34:34.656508 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:34:34.656713 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:34.648273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.3:58137)
0928 02:34:34.648420 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:34.648453 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:34.648910 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:34.649133 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:34.649136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:34.649424 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:34.649601 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:34.649606 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.650241 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.650242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:34.651065 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:34.651067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.651392 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.651393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:34.651512 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:34.651973 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:34.651984 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:34.654204 (+ 2220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:34.655666 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:34.655668 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:34.655670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:34.655881 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:34.656013 (+ 132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:34.656013 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:34.656013 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:34.656090 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:34.656259 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:34.656262 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:34.656364 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 02:34:34.656436 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 02:34:34.656544 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
I20260928 02:34:34.657094 2606 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44123
I20260928 02:34:34.657169 2606 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:34.657356 2606 heartbeater.cc:511] Master 127.0.98.62:44123 requested a full tablet report, sending...
I20260928 02:34:34.657727 2122 ts_manager.cc:194] Registered new tserver with Master: d92c8bf86a9a47bfa8ea1e51855f3b0a (127.0.98.3:36225)
I20260928 02:34:34.658177 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58137
I20260928 02:34:34.658260 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 2475
I20260928 02:34:34.658354 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-2/wal/instance
I20260928 02:34:34.658536 392 external_mini_cluster.cc:1428] Setting key 330a1aa0bd2ec5774394f9f83e266277
I20260928 02:34:34.659292 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:34:34.663339 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:34.657976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.3:54293)
0928 02:34:34.658084 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:34.658094 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:34.658149 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:34.658311 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:34.658312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:34.658495 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:34.658552 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:34.658553 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.659300 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.659301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:34.660045 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:34.660047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.660338 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.660339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:34.660407 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:34.660727 (+ 320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:34.660731 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:34.661053 (+ 322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:34.662521 (+ 1468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:34.662522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:34.662523 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:34.662683 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:34.662861 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:34.662861 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:34.662861 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:34.662903 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:34.663110 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:34.663111 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:34.663150 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 02:34:34.663206 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 02:34:34.663273 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":79,"threads_started":1}
I20260928 02:34:34.663385 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:34.658036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54293 (local address 127.0.98.62:44123)
0928 02:34:34.658097 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:34.658099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:34.658149 (+ 50us) server_negotiation.cc:407] Connection header received
0928 02:34:34.658199 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:34.658200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:34.658209 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:34.658261 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:34.658623 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.659143 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.660113 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.660281 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.660332 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:34.661133 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:34.661135 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:34.661135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:34.661142 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:34.662447 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:34.662745 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:34.662746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:34.662747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:34.662800 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:34.662938 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:34.662938 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:34.662939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:34.663051 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:34.663115 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:34.663243 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 02:34:34.663305 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 02:34:34 dist-test-slave-fz47 krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:34:34 dist-test-slave-fz47 krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:34.680330 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:34.674297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48544 (local address 127.0.98.62:44123)
0928 02:34:34.674341 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:34.674343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:34.674351 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:34:34.674394 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:34.674395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:34.674402 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:34.674449 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:34.674924 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.675390 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.676093 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.676284 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.676348 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:34.678356 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:34.678358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:34.678358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:34.678365 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:34.679425 (+ 1060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:34.679746 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:34.679746 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:34.679747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:34.679787 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:34.679916 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:34.679916 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:34.679916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:34.680008 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:34.680077 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:34.680206 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 02:34:34.680263 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 02:34:34.681818 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:34:34.683274 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:34:34.688989 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:34.685201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40965 (local address 127.0.0.1:52382)
0928 02:34:34.685504 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:34.685548 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:34.685614 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:34.685923 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:34.685924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:34.685926 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:34.685978 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:34.685980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.687010 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.687011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:34.687916 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:34.687918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.688700 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.688701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:34.688776 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:34.688790 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 02:34:34.688866 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 02:34:34.688914 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":30,"thread_start_us":81,"threads_started":1}
I20260928 02:34:34.689018 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:34.685326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52382 (local address 127.0.98.2:40965)
0928 02:34:34.685786 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:34.685790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:34.685799 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:34:34.685837 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:34.685838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:34.685851 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:34.685917 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:34.686069 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.686934 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.688030 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.688623 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.688700 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:34.688767 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:34.688884 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 02:34:34.688922 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":85,"threads_started":1}
I20260928 02:34:34.689419 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:34.685234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36225 (local address 127.0.0.1:36566)
0928 02:34:34.685539 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:34.685549 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:34.685614 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:34.685874 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:34.685877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:34.685880 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:34.685940 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:34.685942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.687214 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.687215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:34.688399 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:34.688401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.689173 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.689174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:34.689272 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:34.689277 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:34:34.689318 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 02:34:34.689353 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":35,"thread_start_us":88,"threads_started":1}
I20260928 02:34:34.689538 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:34.685532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37686 (local address 127.0.98.1:42235)
0928 02:34:34.685693 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:34.685697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:34.685707 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:34:34.685842 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:34.685845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:34.685859 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:34.685930 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:34.686099 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.687218 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.688477 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.689193 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.689324 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:34.689387 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:34.689412 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 02:34:34.689448 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":68,"threads_started":1}
I20260928 02:34:34.689788 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:34.685395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36566 (local address 127.0.98.3:36225)
0928 02:34:34.685565 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:34.685571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:34.685688 (+ 117us) server_negotiation.cc:407] Connection header received
0928 02:34:34.685727 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:34.685729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:34.685744 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:34.685802 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:34.686010 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.687129 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.688488 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.689090 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.689151 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:34.689205 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:34.689687 (+ 482us) server_negotiation.cc:299] Negotiation successful
0928 02:34:34.689721 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
I20260928 02:34:34.689492 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:34.685421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42235 (local address 127.0.0.1:37686)
0928 02:34:34.685610 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:34.685619 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:34.685737 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:34.685995 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:34.685995 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:34.685997 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:34.686028 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:34.686029 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.687311 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.687312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:34.688398 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:34.688401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.689280 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.689281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:34.689344 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:34.689349 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:34:34.689392 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 02:34:34.689429 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":18,"thread_start_us":62,"threads_started":1}
I20260928 02:34:34.689983 2400 tablet_service.cc:1467] Processing CreateTablet for tablet 01e6155186224623ba3de24fbdec75f2 (DEFAULT_TABLE table=test-table [id=0cb8e60f938c4afa96b612ad91ba75a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:34.690356 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e6155186224623ba3de24fbdec75f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:34.690474 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 01e6155186224623ba3de24fbdec75f2 (DEFAULT_TABLE table=test-table [id=0cb8e60f938c4afa96b612ad91ba75a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:34.690670 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 01e6155186224623ba3de24fbdec75f2 (DEFAULT_TABLE table=test-table [id=0cb8e60f938c4afa96b612ad91ba75a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:34.690843 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e6155186224623ba3de24fbdec75f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:34.690958 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e6155186224623ba3de24fbdec75f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:34.694134 2627 tablet_bootstrap.cc:492] T 01e6155186224623ba3de24fbdec75f2 P d92c8bf86a9a47bfa8ea1e51855f3b0a: Bootstrap starting.
I20260928 02:34:34.694389 2626 tablet_bootstrap.cc:492] T 01e6155186224623ba3de24fbdec75f2 P 249e2c930f94405f9f41c1e861b60c53: Bootstrap starting.
I20260928 02:34:34.694913 2626 tablet_bootstrap.cc:654] T 01e6155186224623ba3de24fbdec75f2 P 249e2c930f94405f9f41c1e861b60c53: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:34.695053 2627 tablet_bootstrap.cc:654] T 01e6155186224623ba3de24fbdec75f2 P d92c8bf86a9a47bfa8ea1e51855f3b0a: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:34.695297 2626 log.cc:824] T 01e6155186224623ba3de24fbdec75f2 P 249e2c930f94405f9f41c1e861b60c53: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:34.695490 2627 log.cc:824] T 01e6155186224623ba3de24fbdec75f2 P d92c8bf86a9a47bfa8ea1e51855f3b0a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:34.695657 2628 tablet_bootstrap.cc:492] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a: Bootstrap starting.
I20260928 02:34:34.696308 2628 tablet_bootstrap.cc:654] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:34.696342 2626 tablet_bootstrap.cc:492] T 01e6155186224623ba3de24fbdec75f2 P 249e2c930f94405f9f41c1e861b60c53: No bootstrap required, opened a new log
I20260928 02:34:34.696359 2627 tablet_bootstrap.cc:492] T 01e6155186224623ba3de24fbdec75f2 P d92c8bf86a9a47bfa8ea1e51855f3b0a: No bootstrap required, opened a new log
I20260928 02:34:34.696395 2626 ts_tablet_manager.cc:1397] T 01e6155186224623ba3de24fbdec75f2 P 249e2c930f94405f9f41c1e861b60c53: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:34:34.696425 2627 ts_tablet_manager.cc:1397] T 01e6155186224623ba3de24fbdec75f2 P d92c8bf86a9a47bfa8ea1e51855f3b0a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:34:34.696681 2628 log.cc:824] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:34.697345 2628 tablet_bootstrap.cc:492] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a: No bootstrap required, opened a new log
I20260928 02:34:34.697410 2628 ts_tablet_manager.cc:1397] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:34:34.698357 2626 raft_consensus.cc:348] T 01e6155186224623ba3de24fbdec75f2 P 249e2c930f94405f9f41c1e861b60c53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:34.698529 2626 raft_consensus.cc:374] T 01e6155186224623ba3de24fbdec75f2 P 249e2c930f94405f9f41c1e861b60c53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:34.698562 2626 raft_consensus.cc:729] T 01e6155186224623ba3de24fbdec75f2 P 249e2c930f94405f9f41c1e861b60c53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 249e2c930f94405f9f41c1e861b60c53, State: Initialized, Role: FOLLOWER
I20260928 02:34:34.698669 2626 consensus_queue.cc:260] T 01e6155186224623ba3de24fbdec75f2 P 249e2c930f94405f9f41c1e861b60c53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:34.698612 2627 raft_consensus.cc:348] T 01e6155186224623ba3de24fbdec75f2 P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:34.698770 2627 raft_consensus.cc:374] T 01e6155186224623ba3de24fbdec75f2 P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:34.698804 2627 raft_consensus.cc:729] T 01e6155186224623ba3de24fbdec75f2 P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92c8bf86a9a47bfa8ea1e51855f3b0a, State: Initialized, Role: FOLLOWER
I20260928 02:34:34.698913 2626 ts_tablet_manager.cc:1428] T 01e6155186224623ba3de24fbdec75f2 P 249e2c930f94405f9f41c1e861b60c53: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:34:34.698911 2627 consensus_queue.cc:260] T 01e6155186224623ba3de24fbdec75f2 P d92c8bf86a9a47bfa8ea1e51855f3b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:34.699183 2627 ts_tablet_manager.cc:1428] T 01e6155186224623ba3de24fbdec75f2 P d92c8bf86a9a47bfa8ea1e51855f3b0a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 02:34:34.699240 2466 heartbeater.cc:503] Master 127.0.98.62:44123 was elected leader, sending a full tablet report...
I20260928 02:34:34.699270 2606 heartbeater.cc:503] Master 127.0.98.62:44123 was elected leader, sending a full tablet report...
I20260928 02:34:34.699325 2628 raft_consensus.cc:348] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:34.699525 2628 raft_consensus.cc:374] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:34.699581 2628 raft_consensus.cc:729] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cbf48bcdf314ef8b8c6d89fb9e2150a, State: Initialized, Role: FOLLOWER
I20260928 02:34:34.699687 2628 consensus_queue.cc:260] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:34.699928 2628 ts_tablet_manager.cc:1428] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 02:34:34.699951 2326 heartbeater.cc:503] Master 127.0.98.62:44123 was elected leader, sending a full tablet report...
W20260928 02:34:34.755064 2467 tablet.cc:2369] T 01e6155186224623ba3de24fbdec75f2 P 249e2c930f94405f9f41c1e861b60c53: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:34:34.772179 2634 raft_consensus.cc:482] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:34:34.772305 2634 raft_consensus.cc:504] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:34.772658 2634 leader_election.cc:288] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 249e2c930f94405f9f41c1e861b60c53 (127.0.98.2:40965), d92c8bf86a9a47bfa8ea1e51855f3b0a (127.0.98.3:36225)
I20260928 02:34:34.775358 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:34.772947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57305 (local address 127.0.98.2:40965)
0928 02:34:34.773007 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:34.773009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:34.773018 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:34:34.773044 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:34.773045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:34.773057 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:34.773112 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:34.773270 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.773737 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.774709 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.775059 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.775118 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:34.775129 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:34.775253 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 02:34:34.775293 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 02:34:34.775358 2219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:34.772833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40965 (local address 127.0.98.1:57305)
0928 02:34:34.772897 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:34.772908 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:34.772948 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:34.773119 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:34.773120 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:34.773122 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:34.773187 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:34.773188 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.773810 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.773811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:34.774619 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:34.774621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.775129 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.775129 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:34.775192 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:34.775195 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:34:34.775238 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 02:34:34.775277 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 02:34:34.775439 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:34.773018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59005 (local address 127.0.98.3:36225)
0928 02:34:34.773069 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:34.773071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:34.773079 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:34:34.773129 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:34.773130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:34.773141 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:34.773184 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:34.773351 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.773821 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.774762 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.775125 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.775182 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:34.775193 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:34.775338 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 02:34:34.775372 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 02:34:34.775465 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:34.772850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36225 (local address 127.0.98.1:59005)
0928 02:34:34.773030 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:34.773039 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:34.773081 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:34.773197 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:34.773198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:34.773199 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:34.773289 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:34.773290 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.773886 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.773887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:34.774675 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:34.774677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.775197 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.775198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:34.775277 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:34.775281 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:34:34.775325 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 02:34:34.775362 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":86,"threads_started":1}
I20260928 02:34:34.775756 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e6155186224623ba3de24fbdec75f2" candidate_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" is_pre_election: true
I20260928 02:34:34.775763 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e6155186224623ba3de24fbdec75f2" candidate_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "249e2c930f94405f9f41c1e861b60c53" is_pre_election: true
I20260928 02:34:34.775902 2420 raft_consensus.cc:2454] T 01e6155186224623ba3de24fbdec75f2 P 249e2c930f94405f9f41c1e861b60c53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cbf48bcdf314ef8b8c6d89fb9e2150a in term 0.
I20260928 02:34:34.775903 2560 raft_consensus.cc:2454] T 01e6155186224623ba3de24fbdec75f2 P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cbf48bcdf314ef8b8c6d89fb9e2150a in term 0.
I20260928 02:34:34.776100 2214 leader_election.cc:302] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cbf48bcdf314ef8b8c6d89fb9e2150a, d92c8bf86a9a47bfa8ea1e51855f3b0a; no voters:
I20260928 02:34:34.776208 2634 raft_consensus.cc:2781] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:34:34.776264 2634 raft_consensus.cc:482] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:34:34.776280 2634 raft_consensus.cc:3037] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:34.776906 2634 raft_consensus.cc:504] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:34.777014 2634 leader_election.cc:288] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [CANDIDATE]: Term 1 election: Requested vote from peers 249e2c930f94405f9f41c1e861b60c53 (127.0.98.2:40965), d92c8bf86a9a47bfa8ea1e51855f3b0a (127.0.98.3:36225)
I20260928 02:34:34.777146 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e6155186224623ba3de24fbdec75f2" candidate_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a"
I20260928 02:34:34.777156 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e6155186224623ba3de24fbdec75f2" candidate_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "249e2c930f94405f9f41c1e861b60c53"
I20260928 02:34:34.777221 2560 raft_consensus.cc:3037] T 01e6155186224623ba3de24fbdec75f2 P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:34.777223 2420 raft_consensus.cc:3037] T 01e6155186224623ba3de24fbdec75f2 P 249e2c930f94405f9f41c1e861b60c53 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:34.777868 2560 raft_consensus.cc:2454] T 01e6155186224623ba3de24fbdec75f2 P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cbf48bcdf314ef8b8c6d89fb9e2150a in term 1.
I20260928 02:34:34.777931 2420 raft_consensus.cc:2454] T 01e6155186224623ba3de24fbdec75f2 P 249e2c930f94405f9f41c1e861b60c53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cbf48bcdf314ef8b8c6d89fb9e2150a in term 1.
I20260928 02:34:34.778012 2214 leader_election.cc:302] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cbf48bcdf314ef8b8c6d89fb9e2150a, d92c8bf86a9a47bfa8ea1e51855f3b0a; no voters:
I20260928 02:34:34.778101 2634 raft_consensus.cc:2781] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:34.778226 2634 raft_consensus.cc:686] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 1 LEADER]: Becoming Leader. State: Replica: 2cbf48bcdf314ef8b8c6d89fb9e2150a, State: Running, Role: LEADER
I20260928 02:34:34.778303 2634 consensus_queue.cc:237] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:34.779076 2122 catalog_manager.cc:5495] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cbf48bcdf314ef8b8c6d89fb9e2150a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } health_report { overall_health: UNKNOWN } } }
Sep 28 02:34:34 dist-test-slave-fz47 krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:34.811781 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:34.805862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48558 (local address 127.0.98.62:44123)
0928 02:34:34.805911 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:34.805914 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:34.806022 (+ 108us) server_negotiation.cc:407] Connection header received
0928 02:34:34.806075 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:34.806076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:34.806085 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:34.806129 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:34.806575 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.807039 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.807759 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.807928 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.807979 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:34.809850 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:34.809851 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:34.809851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:34.809857 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:34.810925 (+ 1068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:34.811226 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:34.811227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:34.811227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:34.811285 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:34.811415 (+ 130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:34.811415 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:34.811415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:34.811501 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:34.811543 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:34.811641 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 02:34:34.811700 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 02:34:34.811854 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:34.805815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.0.1:48558)
0928 02:34:34.805959 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:34.805968 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:34.806030 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:34.806138 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:34.806138 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:34.806458 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:34.806508 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:34.806510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.807122 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.807122 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:34.807688 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:34.807690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.807981 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.807982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:34.808028 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:34.808522 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:34.808527 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:34.809778 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:34.810989 (+ 1211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:34.810994 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:34.810996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:34.811173 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:34.811331 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:34.811331 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:34.811331 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:34.811378 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:34.811539 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:34.811542 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:34.811582 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 02:34:34.811625 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 02:34:34.811745 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":44,"thread_start_us":72,"threads_started":1}
I20260928 02:34:34.813489 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48558:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:34:34.813658 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:34:34.814936 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 787aac3978004ecbbff5d4fa8d85f9c5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7ee18161f0af4457bd9d5847f8fddad6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:34:34.814929 2400 tablet_service.cc:1467] Processing CreateTablet for tablet 787aac3978004ecbbff5d4fa8d85f9c5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7ee18161f0af4457bd9d5847f8fddad6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:34:34.815033 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 787aac3978004ecbbff5d4fa8d85f9c5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7ee18161f0af4457bd9d5847f8fddad6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:34:34.815080 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 787aac3978004ecbbff5d4fa8d85f9c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:34.815080 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 787aac3978004ecbbff5d4fa8d85f9c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:34.815160 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 787aac3978004ecbbff5d4fa8d85f9c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:34.816107 2627 tablet_bootstrap.cc:492] T 787aac3978004ecbbff5d4fa8d85f9c5 P d92c8bf86a9a47bfa8ea1e51855f3b0a: Bootstrap starting.
I20260928 02:34:34.816498 2628 tablet_bootstrap.cc:492] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a: Bootstrap starting.
I20260928 02:34:34.816581 2627 tablet_bootstrap.cc:654] T 787aac3978004ecbbff5d4fa8d85f9c5 P d92c8bf86a9a47bfa8ea1e51855f3b0a: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:34.816744 2626 tablet_bootstrap.cc:492] T 787aac3978004ecbbff5d4fa8d85f9c5 P 249e2c930f94405f9f41c1e861b60c53: Bootstrap starting.
I20260928 02:34:34.816896 2628 tablet_bootstrap.cc:654] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:34.817281 2627 tablet_bootstrap.cc:492] T 787aac3978004ecbbff5d4fa8d85f9c5 P d92c8bf86a9a47bfa8ea1e51855f3b0a: No bootstrap required, opened a new log
I20260928 02:34:34.817328 2627 ts_tablet_manager.cc:1397] T 787aac3978004ecbbff5d4fa8d85f9c5 P d92c8bf86a9a47bfa8ea1e51855f3b0a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 02:34:34.817365 2626 tablet_bootstrap.cc:654] T 787aac3978004ecbbff5d4fa8d85f9c5 P 249e2c930f94405f9f41c1e861b60c53: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:34.817458 2627 raft_consensus.cc:348] T 787aac3978004ecbbff5d4fa8d85f9c5 P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:34.817515 2627 raft_consensus.cc:374] T 787aac3978004ecbbff5d4fa8d85f9c5 P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:34.817529 2627 raft_consensus.cc:729] T 787aac3978004ecbbff5d4fa8d85f9c5 P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92c8bf86a9a47bfa8ea1e51855f3b0a, State: Initialized, Role: FOLLOWER
I20260928 02:34:34.817569 2628 tablet_bootstrap.cc:492] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a: No bootstrap required, opened a new log
I20260928 02:34:34.817562 2627 consensus_queue.cc:260] T 787aac3978004ecbbff5d4fa8d85f9c5 P d92c8bf86a9a47bfa8ea1e51855f3b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:34.817611 2628 ts_tablet_manager.cc:1397] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 02:34:34.817639 2627 ts_tablet_manager.cc:1428] T 787aac3978004ecbbff5d4fa8d85f9c5 P d92c8bf86a9a47bfa8ea1e51855f3b0a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:34.817688 2633 tablet_replica.cc:439] T 787aac3978004ecbbff5d4fa8d85f9c5 P d92c8bf86a9a47bfa8ea1e51855f3b0a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } } }
I20260928 02:34:34.817741 2628 raft_consensus.cc:348] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:34.817780 2633 tablet_replica.cc:442] T 787aac3978004ecbbff5d4fa8d85f9c5 P d92c8bf86a9a47bfa8ea1e51855f3b0a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:34:34.817812 2628 raft_consensus.cc:374] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:34.817831 2628 raft_consensus.cc:729] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cbf48bcdf314ef8b8c6d89fb9e2150a, State: Initialized, Role: FOLLOWER
I20260928 02:34:34.817865 2628 consensus_queue.cc:260] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:34.817940 2628 ts_tablet_manager.cc:1428] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:34.818044 2634 tablet_replica.cc:439] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } } }
I20260928 02:34:34.818120 2634 tablet_replica.cc:442] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:34:34.818219 2626 tablet_bootstrap.cc:492] T 787aac3978004ecbbff5d4fa8d85f9c5 P 249e2c930f94405f9f41c1e861b60c53: No bootstrap required, opened a new log
I20260928 02:34:34.818269 2626 ts_tablet_manager.cc:1397] T 787aac3978004ecbbff5d4fa8d85f9c5 P 249e2c930f94405f9f41c1e861b60c53: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 02:34:34.818414 2626 raft_consensus.cc:348] T 787aac3978004ecbbff5d4fa8d85f9c5 P 249e2c930f94405f9f41c1e861b60c53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:34.818490 2626 raft_consensus.cc:374] T 787aac3978004ecbbff5d4fa8d85f9c5 P 249e2c930f94405f9f41c1e861b60c53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:34.818503 2626 raft_consensus.cc:729] T 787aac3978004ecbbff5d4fa8d85f9c5 P 249e2c930f94405f9f41c1e861b60c53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 249e2c930f94405f9f41c1e861b60c53, State: Initialized, Role: FOLLOWER
I20260928 02:34:34.818537 2626 consensus_queue.cc:260] T 787aac3978004ecbbff5d4fa8d85f9c5 P 249e2c930f94405f9f41c1e861b60c53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:34.818615 2626 ts_tablet_manager.cc:1428] T 787aac3978004ecbbff5d4fa8d85f9c5 P 249e2c930f94405f9f41c1e861b60c53: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:34.818678 2632 tablet_replica.cc:439] T 787aac3978004ecbbff5d4fa8d85f9c5 P 249e2c930f94405f9f41c1e861b60c53: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } } }
I20260928 02:34:34.818768 2632 tablet_replica.cc:442] T 787aac3978004ecbbff5d4fa8d85f9c5 P 249e2c930f94405f9f41c1e861b60c53: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:34:34.853317 2634 raft_consensus.cc:482] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:34:34.853380 2634 raft_consensus.cc:504] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:34.853487 2634 leader_election.cc:288] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 249e2c930f94405f9f41c1e861b60c53 (127.0.98.2:40965), d92c8bf86a9a47bfa8ea1e51855f3b0a (127.0.98.3:36225)
I20260928 02:34:34.853654 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "787aac3978004ecbbff5d4fa8d85f9c5" candidate_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" is_pre_election: true
I20260928 02:34:34.853643 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "787aac3978004ecbbff5d4fa8d85f9c5" candidate_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "249e2c930f94405f9f41c1e861b60c53" is_pre_election: true
I20260928 02:34:34.853730 2420 raft_consensus.cc:2454] T 787aac3978004ecbbff5d4fa8d85f9c5 P 249e2c930f94405f9f41c1e861b60c53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cbf48bcdf314ef8b8c6d89fb9e2150a in term 0.
I20260928 02:34:34.853730 2560 raft_consensus.cc:2454] T 787aac3978004ecbbff5d4fa8d85f9c5 P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cbf48bcdf314ef8b8c6d89fb9e2150a in term 0.
I20260928 02:34:34.853873 2212 leader_election.cc:302] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 249e2c930f94405f9f41c1e861b60c53, 2cbf48bcdf314ef8b8c6d89fb9e2150a; no voters:
I20260928 02:34:34.853979 2634 raft_consensus.cc:2781] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:34:34.854017 2634 raft_consensus.cc:482] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:34:34.854030 2634 raft_consensus.cc:3037] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:34.854559 2634 raft_consensus.cc:504] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:34.854655 2634 leader_election.cc:288] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [CANDIDATE]: Term 1 election: Requested vote from peers 249e2c930f94405f9f41c1e861b60c53 (127.0.98.2:40965), d92c8bf86a9a47bfa8ea1e51855f3b0a (127.0.98.3:36225)
I20260928 02:34:34.854802 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "787aac3978004ecbbff5d4fa8d85f9c5" candidate_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "249e2c930f94405f9f41c1e861b60c53"
I20260928 02:34:34.854794 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "787aac3978004ecbbff5d4fa8d85f9c5" candidate_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a"
I20260928 02:34:34.854866 2420 raft_consensus.cc:3037] T 787aac3978004ecbbff5d4fa8d85f9c5 P 249e2c930f94405f9f41c1e861b60c53 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:34.854866 2560 raft_consensus.cc:3037] T 787aac3978004ecbbff5d4fa8d85f9c5 P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:34.855389 2420 raft_consensus.cc:2454] T 787aac3978004ecbbff5d4fa8d85f9c5 P 249e2c930f94405f9f41c1e861b60c53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cbf48bcdf314ef8b8c6d89fb9e2150a in term 1.
I20260928 02:34:34.855450 2560 raft_consensus.cc:2454] T 787aac3978004ecbbff5d4fa8d85f9c5 P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cbf48bcdf314ef8b8c6d89fb9e2150a in term 1.
I20260928 02:34:34.855536 2212 leader_election.cc:302] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 249e2c930f94405f9f41c1e861b60c53, 2cbf48bcdf314ef8b8c6d89fb9e2150a; no voters:
I20260928 02:34:34.855611 2634 raft_consensus.cc:2781] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:34.855664 2634 raft_consensus.cc:686] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 1 LEADER]: Becoming Leader. State: Replica: 2cbf48bcdf314ef8b8c6d89fb9e2150a, State: Running, Role: LEADER
I20260928 02:34:34.855708 2634 consensus_queue.cc:237] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:34.855978 2636 tablet_replica.cc:439] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a: TxnStatusTablet state changed. Reason: New leader 2cbf48bcdf314ef8b8c6d89fb9e2150a. Latest consensus state: current_term: 1 leader_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } } }
I20260928 02:34:34.856036 2636 tablet_replica.cc:442] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:34:34.856213 2648 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:34:34.856230 2122 catalog_manager.cc:5495] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cbf48bcdf314ef8b8c6d89fb9e2150a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } health_report { overall_health: UNKNOWN } } }
I20260928 02:34:34.856340 2648 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20260928 02:34:34.864629 2327 tablet.cc:2369] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:34:34.865113 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:34.862463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37700 (local address 127.0.98.1:42235)
0928 02:34:34.862509 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:34.862511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:34.862519 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:34:34.862584 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:34.862584 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:34.862599 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:34.862650 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:34:34.862758 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.863217 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.864291 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.864498 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.864581 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:34.864792 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:34:34.864901 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:34:34.864982 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:34.865050 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 02:34:34.865069 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 02:34:34.865170 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:34.862382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42235 (local address 127.0.0.1:37700)
0928 02:34:34.862456 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:34.862469 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:34.862518 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:34.862655 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:34.862655 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:34.862657 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:34:34.862703 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:34.862706 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.863281 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.863282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:34.864194 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:34.864197 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.864586 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.864587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:34.864653 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:34.864669 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:34:34.864991 (+ 322us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:34:34.865000 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 02:34:34.865047 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 02:34:34.865098 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 02:34:34.866948 2560 raft_consensus.cc:1261] T 787aac3978004ecbbff5d4fa8d85f9c5 P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 1 FOLLOWER]: Refusing update from remote peer 2cbf48bcdf314ef8b8c6d89fb9e2150a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:34.867219 2420 raft_consensus.cc:1261] T 787aac3978004ecbbff5d4fa8d85f9c5 P 249e2c930f94405f9f41c1e861b60c53 [term 1 FOLLOWER]: Refusing update from remote peer 2cbf48bcdf314ef8b8c6d89fb9e2150a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:34.867264 2634 consensus_queue.cc:1035] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:34.867465 2634 consensus_queue.cc:1035] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [LEADER]: Connected to new peer: Peer: permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:34.868549 2633 tablet_replica.cc:439] T 787aac3978004ecbbff5d4fa8d85f9c5 P d92c8bf86a9a47bfa8ea1e51855f3b0a: TxnStatusTablet state changed. Reason: New leader 2cbf48bcdf314ef8b8c6d89fb9e2150a. Latest consensus state: current_term: 1 leader_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } } }
I20260928 02:34:34.868644 2633 tablet_replica.cc:442] T 787aac3978004ecbbff5d4fa8d85f9c5 P d92c8bf86a9a47bfa8ea1e51855f3b0a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:34:34.869412 2632 tablet_replica.cc:439] T 787aac3978004ecbbff5d4fa8d85f9c5 P 249e2c930f94405f9f41c1e861b60c53: TxnStatusTablet state changed. Reason: New leader 2cbf48bcdf314ef8b8c6d89fb9e2150a. Latest consensus state: current_term: 1 leader_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } } }
I20260928 02:34:34.869511 2632 tablet_replica.cc:442] T 787aac3978004ecbbff5d4fa8d85f9c5 P 249e2c930f94405f9f41c1e861b60c53: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:34:34.870153 2636 tablet_replica.cc:439] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } } }
I20260928 02:34:34.870230 2636 tablet_replica.cc:442] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:34:34.870746 2654 mvcc.cc:205] Tried to move back new op lower bound from 7334145535452385280 to 7334145535409446912. Current Snapshot: MvccSnapshot[applied={T|T < 7334145535452385280}]
I20260928 02:34:34.870775 2634 tablet_replica.cc:439] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } } }
I20260928 02:34:34.870857 2634 tablet_replica.cc:442] T 787aac3978004ecbbff5d4fa8d85f9c5 P 2cbf48bcdf314ef8b8c6d89fb9e2150a: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:34:34.870863 2632 tablet_replica.cc:439] T 787aac3978004ecbbff5d4fa8d85f9c5 P 249e2c930f94405f9f41c1e861b60c53: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } } }
I20260928 02:34:34.870930 2632 tablet_replica.cc:442] T 787aac3978004ecbbff5d4fa8d85f9c5 P 249e2c930f94405f9f41c1e861b60c53: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:34:34.872620 2633 tablet_replica.cc:439] T 787aac3978004ecbbff5d4fa8d85f9c5 P d92c8bf86a9a47bfa8ea1e51855f3b0a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } } }
I20260928 02:34:34.872697 2633 tablet_replica.cc:442] T 787aac3978004ecbbff5d4fa8d85f9c5 P d92c8bf86a9a47bfa8ea1e51855f3b0a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:34:34.872908 2652 mvcc.cc:205] Tried to move back new op lower bound from 7334145535452385280 to 7334145535409446912. Current Snapshot: MvccSnapshot[applied={T|T < 7334145535452385280 or (T in {7334145535452385280})}]
I20260928 02:34:34.874549 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:34.872092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37712 (local address 127.0.98.1:42235)
0928 02:34:34.872201 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:34.872203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:34.872372 (+ 169us) server_negotiation.cc:407] Connection header received
0928 02:34:34.872390 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:34.872391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:34.872414 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:34.872452 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:34:34.872593 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.873231 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.873979 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.874177 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.874230 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:34.874346 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:34:34.874378 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:34:34.874422 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:34.874471 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 02:34:34.874488 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 02:34:34.879343 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:34.876783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35051 (local address 127.0.98.1:42235)
0928 02:34:34.876821 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:34.876822 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:34.876938 (+ 116us) server_negotiation.cc:407] Connection header received
0928 02:34:34.876961 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:34.876962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:34.876970 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:34.877025 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:34:34.877168 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.877901 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.878616 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.878911 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.878977 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:34.879113 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:34:34.879142 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:34:34.879188 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:34.879229 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 02:34:34.879244 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:34:34.879360 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:34.876727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42235 (local address 127.0.98.1:35051)
0928 02:34:34.876874 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:34.876882 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:34.876930 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:34.877044 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:34.877045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:34.877055 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:34:34.877107 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:34.877108 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.877971 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.877972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:34.878550 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:34.878551 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:34.878976 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:34.878976 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:34.879030 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:34.879042 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:34:34.879192 (+ 150us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:34:34.879193 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 02:34:34.879224 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 02:34:34.879255 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":67,"threads_started":1}
I20260928 02:34:34.881462 2420 raft_consensus.cc:1261] T 01e6155186224623ba3de24fbdec75f2 P 249e2c930f94405f9f41c1e861b60c53 [term 1 FOLLOWER]: Refusing update from remote peer 2cbf48bcdf314ef8b8c6d89fb9e2150a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:34.881487 2560 raft_consensus.cc:1261] T 01e6155186224623ba3de24fbdec75f2 P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 1 FOLLOWER]: Refusing update from remote peer 2cbf48bcdf314ef8b8c6d89fb9e2150a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:34.881702 2636 consensus_queue.cc:1035] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:34.881796 2634 consensus_queue.cc:1035] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [LEADER]: Connected to new peer: Peer: permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:34.885983 2651 mvcc.cc:205] Tried to move back new op lower bound from 7334145535513378816 to 7334145535092609024. Current Snapshot: MvccSnapshot[applied={T|T < 7334145535513378816}]
I20260928 02:34:34.898273 2123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:34:34.898357 2123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:34:34.898983 2123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d1911e566f74fc9b0af803869790bcc: Sending DeleteTablet for 3 replicas of tablet 01e6155186224623ba3de24fbdec75f2
I20260928 02:34:34.899278 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet 01e6155186224623ba3de24fbdec75f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52382
I20260928 02:34:34.899323 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 01e6155186224623ba3de24fbdec75f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37686
I20260928 02:34:34.899503 2667 tablet_replica.cc:330] T 01e6155186224623ba3de24fbdec75f2 P 249e2c930f94405f9f41c1e861b60c53: stopping tablet replica
I20260928 02:34:34.899616 2668 tablet_replica.cc:330] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a: stopping tablet replica
I20260928 02:34:34.899638 2667 raft_consensus.cc:2229] T 01e6155186224623ba3de24fbdec75f2 P 249e2c930f94405f9f41c1e861b60c53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:34:34.899612 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 01e6155186224623ba3de24fbdec75f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36566
I20260928 02:34:34.899715 2667 raft_consensus.cc:2258] T 01e6155186224623ba3de24fbdec75f2 P 249e2c930f94405f9f41c1e861b60c53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:34.899715 2668 raft_consensus.cc:2229] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:34:34.899816 2668 raft_consensus.cc:2258] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:34.899932 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:34:34.900074 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:34:34.900197 2667 ts_tablet_manager.cc:1905] T 01e6155186224623ba3de24fbdec75f2 P 249e2c930f94405f9f41c1e861b60c53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:34.900704 2668 ts_tablet_manager.cc:1905] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:34.901324 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 647f4453ea154a6f9cecefeb1e5dc39d (DEFAULT_TABLE table=test-table [id=2f53679cd36b4218b701d3c16c93321e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:34.901396 2400 tablet_service.cc:1467] Processing CreateTablet for tablet 647f4453ea154a6f9cecefeb1e5dc39d (DEFAULT_TABLE table=test-table [id=2f53679cd36b4218b701d3c16c93321e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:34.901424 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 647f4453ea154a6f9cecefeb1e5dc39d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:34.901494 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 647f4453ea154a6f9cecefeb1e5dc39d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:34.901921 2667 ts_tablet_manager.cc:1918] T 01e6155186224623ba3de24fbdec75f2 P 249e2c930f94405f9f41c1e861b60c53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:34:34.901978 2667 log.cc:1196] T 01e6155186224623ba3de24fbdec75f2 P 249e2c930f94405f9f41c1e861b60c53: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-1/wal/wals/01e6155186224623ba3de24fbdec75f2
I20260928 02:34:34.902205 2667 ts_tablet_manager.cc:1939] T 01e6155186224623ba3de24fbdec75f2 P 249e2c930f94405f9f41c1e861b60c53: Deleting consensus metadata
I20260928 02:34:34.902606 2626 tablet_bootstrap.cc:492] T 647f4453ea154a6f9cecefeb1e5dc39d P 249e2c930f94405f9f41c1e861b60c53: Bootstrap starting.
I20260928 02:34:34.902648 2628 tablet_bootstrap.cc:492] T 647f4453ea154a6f9cecefeb1e5dc39d P 2cbf48bcdf314ef8b8c6d89fb9e2150a: Bootstrap starting.
I20260928 02:34:34.902690 2108 catalog_manager.cc:4841] TS 249e2c930f94405f9f41c1e861b60c53 (127.0.98.2:40965): tablet 01e6155186224623ba3de24fbdec75f2 (table test-table [id=0cb8e60f938c4afa96b612ad91ba75a6]) successfully deleted
I20260928 02:34:34.902901 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 647f4453ea154a6f9cecefeb1e5dc39d (DEFAULT_TABLE table=test-table [id=2f53679cd36b4218b701d3c16c93321e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:34.902971 2626 tablet_bootstrap.cc:654] T 647f4453ea154a6f9cecefeb1e5dc39d P 249e2c930f94405f9f41c1e861b60c53: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:34.902989 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 647f4453ea154a6f9cecefeb1e5dc39d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:34.903015 2628 tablet_bootstrap.cc:654] T 647f4453ea154a6f9cecefeb1e5dc39d P 2cbf48bcdf314ef8b8c6d89fb9e2150a: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:34.903746 2668 ts_tablet_manager.cc:1918] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:34:34.903806 2668 log.cc:1196] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-0/wal/wals/01e6155186224623ba3de24fbdec75f2
I20260928 02:34:34.904034 2668 ts_tablet_manager.cc:1939] T 01e6155186224623ba3de24fbdec75f2 P 2cbf48bcdf314ef8b8c6d89fb9e2150a: Deleting consensus metadata
I20260928 02:34:34.904245 2627 tablet_bootstrap.cc:492] T 647f4453ea154a6f9cecefeb1e5dc39d P d92c8bf86a9a47bfa8ea1e51855f3b0a: Bootstrap starting.
I20260928 02:34:34.904486 2108 catalog_manager.cc:4841] TS 2cbf48bcdf314ef8b8c6d89fb9e2150a (127.0.98.1:42235): tablet 01e6155186224623ba3de24fbdec75f2 (table test-table [id=0cb8e60f938c4afa96b612ad91ba75a6]) successfully deleted
I20260928 02:34:34.904507 2669 tablet_replica.cc:330] T 01e6155186224623ba3de24fbdec75f2 P d92c8bf86a9a47bfa8ea1e51855f3b0a: stopping tablet replica
I20260928 02:34:34.904601 2669 raft_consensus.cc:2229] T 01e6155186224623ba3de24fbdec75f2 P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:34:34.904613 2627 tablet_bootstrap.cc:654] T 647f4453ea154a6f9cecefeb1e5dc39d P d92c8bf86a9a47bfa8ea1e51855f3b0a: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:34.904646 2669 raft_consensus.cc:2258] T 01e6155186224623ba3de24fbdec75f2 P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:34.905157 2627 tablet_bootstrap.cc:492] T 647f4453ea154a6f9cecefeb1e5dc39d P d92c8bf86a9a47bfa8ea1e51855f3b0a: No bootstrap required, opened a new log
I20260928 02:34:34.905205 2627 ts_tablet_manager.cc:1397] T 647f4453ea154a6f9cecefeb1e5dc39d P d92c8bf86a9a47bfa8ea1e51855f3b0a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:34:34.905344 2626 tablet_bootstrap.cc:492] T 647f4453ea154a6f9cecefeb1e5dc39d P 249e2c930f94405f9f41c1e861b60c53: No bootstrap required, opened a new log
I20260928 02:34:34.905356 2627 raft_consensus.cc:348] T 647f4453ea154a6f9cecefeb1e5dc39d P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:34.905390 2626 ts_tablet_manager.cc:1397] T 647f4453ea154a6f9cecefeb1e5dc39d P 249e2c930f94405f9f41c1e861b60c53: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 02:34:34.905417 2627 raft_consensus.cc:374] T 647f4453ea154a6f9cecefeb1e5dc39d P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:34.905436 2627 raft_consensus.cc:729] T 647f4453ea154a6f9cecefeb1e5dc39d P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92c8bf86a9a47bfa8ea1e51855f3b0a, State: Initialized, Role: FOLLOWER
I20260928 02:34:34.905475 2627 consensus_queue.cc:260] T 647f4453ea154a6f9cecefeb1e5dc39d P d92c8bf86a9a47bfa8ea1e51855f3b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:34.905551 2627 ts_tablet_manager.cc:1428] T 647f4453ea154a6f9cecefeb1e5dc39d P d92c8bf86a9a47bfa8ea1e51855f3b0a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:34.905519 2626 raft_consensus.cc:348] T 647f4453ea154a6f9cecefeb1e5dc39d P 249e2c930f94405f9f41c1e861b60c53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:34.905580 2626 raft_consensus.cc:374] T 647f4453ea154a6f9cecefeb1e5dc39d P 249e2c930f94405f9f41c1e861b60c53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:34.905606 2626 raft_consensus.cc:729] T 647f4453ea154a6f9cecefeb1e5dc39d P 249e2c930f94405f9f41c1e861b60c53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 249e2c930f94405f9f41c1e861b60c53, State: Initialized, Role: FOLLOWER
I20260928 02:34:34.905640 2626 consensus_queue.cc:260] T 647f4453ea154a6f9cecefeb1e5dc39d P 249e2c930f94405f9f41c1e861b60c53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:34.905136 2669 ts_tablet_manager.cc:1905] T 01e6155186224623ba3de24fbdec75f2 P d92c8bf86a9a47bfa8ea1e51855f3b0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:34.905717 2626 ts_tablet_manager.cc:1428] T 647f4453ea154a6f9cecefeb1e5dc39d P 249e2c930f94405f9f41c1e861b60c53: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:34.906977 2669 ts_tablet_manager.cc:1918] T 01e6155186224623ba3de24fbdec75f2 P d92c8bf86a9a47bfa8ea1e51855f3b0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:34:34.907032 2669 log.cc:1196] T 01e6155186224623ba3de24fbdec75f2 P d92c8bf86a9a47bfa8ea1e51855f3b0a: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-2/wal/wals/01e6155186224623ba3de24fbdec75f2
I20260928 02:34:34.907227 2669 ts_tablet_manager.cc:1939] T 01e6155186224623ba3de24fbdec75f2 P d92c8bf86a9a47bfa8ea1e51855f3b0a: Deleting consensus metadata
I20260928 02:34:34.907678 2110 catalog_manager.cc:4841] TS d92c8bf86a9a47bfa8ea1e51855f3b0a (127.0.98.3:36225): tablet 01e6155186224623ba3de24fbdec75f2 (table test-table [id=0cb8e60f938c4afa96b612ad91ba75a6]) successfully deleted
I20260928 02:34:34.913249 2628 tablet_bootstrap.cc:492] T 647f4453ea154a6f9cecefeb1e5dc39d P 2cbf48bcdf314ef8b8c6d89fb9e2150a: No bootstrap required, opened a new log
I20260928 02:34:34.913321 2628 ts_tablet_manager.cc:1397] T 647f4453ea154a6f9cecefeb1e5dc39d P 2cbf48bcdf314ef8b8c6d89fb9e2150a: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.001s
I20260928 02:34:34.913460 2628 raft_consensus.cc:348] T 647f4453ea154a6f9cecefeb1e5dc39d P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:34.913511 2628 raft_consensus.cc:374] T 647f4453ea154a6f9cecefeb1e5dc39d P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:34.913522 2628 raft_consensus.cc:729] T 647f4453ea154a6f9cecefeb1e5dc39d P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cbf48bcdf314ef8b8c6d89fb9e2150a, State: Initialized, Role: FOLLOWER
I20260928 02:34:34.913555 2628 consensus_queue.cc:260] T 647f4453ea154a6f9cecefeb1e5dc39d P 2cbf48bcdf314ef8b8c6d89fb9e2150a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:34.913648 2628 ts_tablet_manager.cc:1428] T 647f4453ea154a6f9cecefeb1e5dc39d P 2cbf48bcdf314ef8b8c6d89fb9e2150a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:35.129495 2665 raft_consensus.cc:482] T 647f4453ea154a6f9cecefeb1e5dc39d P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:34:35.129580 2665 raft_consensus.cc:504] T 647f4453ea154a6f9cecefeb1e5dc39d P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:35.129711 2665 leader_election.cc:288] T 647f4453ea154a6f9cecefeb1e5dc39d P 2cbf48bcdf314ef8b8c6d89fb9e2150a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 249e2c930f94405f9f41c1e861b60c53 (127.0.98.2:40965), d92c8bf86a9a47bfa8ea1e51855f3b0a (127.0.98.3:36225)
I20260928 02:34:35.129895 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647f4453ea154a6f9cecefeb1e5dc39d" candidate_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "249e2c930f94405f9f41c1e861b60c53" is_pre_election: true
I20260928 02:34:35.129904 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647f4453ea154a6f9cecefeb1e5dc39d" candidate_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" is_pre_election: true
I20260928 02:34:35.130003 2560 raft_consensus.cc:2454] T 647f4453ea154a6f9cecefeb1e5dc39d P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cbf48bcdf314ef8b8c6d89fb9e2150a in term 0.
I20260928 02:34:35.130003 2420 raft_consensus.cc:2454] T 647f4453ea154a6f9cecefeb1e5dc39d P 249e2c930f94405f9f41c1e861b60c53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cbf48bcdf314ef8b8c6d89fb9e2150a in term 0.
I20260928 02:34:35.130134 2214 leader_election.cc:302] T 647f4453ea154a6f9cecefeb1e5dc39d P 2cbf48bcdf314ef8b8c6d89fb9e2150a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cbf48bcdf314ef8b8c6d89fb9e2150a, d92c8bf86a9a47bfa8ea1e51855f3b0a; no voters:
I20260928 02:34:35.130242 2665 raft_consensus.cc:2781] T 647f4453ea154a6f9cecefeb1e5dc39d P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:34:35.130291 2665 raft_consensus.cc:482] T 647f4453ea154a6f9cecefeb1e5dc39d P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:34:35.130304 2665 raft_consensus.cc:3037] T 647f4453ea154a6f9cecefeb1e5dc39d P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:35.130931 2665 raft_consensus.cc:504] T 647f4453ea154a6f9cecefeb1e5dc39d P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:35.131044 2665 leader_election.cc:288] T 647f4453ea154a6f9cecefeb1e5dc39d P 2cbf48bcdf314ef8b8c6d89fb9e2150a [CANDIDATE]: Term 1 election: Requested vote from peers 249e2c930f94405f9f41c1e861b60c53 (127.0.98.2:40965), d92c8bf86a9a47bfa8ea1e51855f3b0a (127.0.98.3:36225)
I20260928 02:34:35.131177 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647f4453ea154a6f9cecefeb1e5dc39d" candidate_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a"
I20260928 02:34:35.131187 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "647f4453ea154a6f9cecefeb1e5dc39d" candidate_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "249e2c930f94405f9f41c1e861b60c53"
I20260928 02:34:35.131249 2420 raft_consensus.cc:3037] T 647f4453ea154a6f9cecefeb1e5dc39d P 249e2c930f94405f9f41c1e861b60c53 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:35.131245 2560 raft_consensus.cc:3037] T 647f4453ea154a6f9cecefeb1e5dc39d P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:35.131779 2560 raft_consensus.cc:2454] T 647f4453ea154a6f9cecefeb1e5dc39d P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cbf48bcdf314ef8b8c6d89fb9e2150a in term 1.
I20260928 02:34:35.131830 2420 raft_consensus.cc:2454] T 647f4453ea154a6f9cecefeb1e5dc39d P 249e2c930f94405f9f41c1e861b60c53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cbf48bcdf314ef8b8c6d89fb9e2150a in term 1.
I20260928 02:34:35.131942 2214 leader_election.cc:302] T 647f4453ea154a6f9cecefeb1e5dc39d P 2cbf48bcdf314ef8b8c6d89fb9e2150a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cbf48bcdf314ef8b8c6d89fb9e2150a, d92c8bf86a9a47bfa8ea1e51855f3b0a; no voters:
I20260928 02:34:35.132054 2665 raft_consensus.cc:2781] T 647f4453ea154a6f9cecefeb1e5dc39d P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:35.132114 2665 raft_consensus.cc:686] T 647f4453ea154a6f9cecefeb1e5dc39d P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 1 LEADER]: Becoming Leader. State: Replica: 2cbf48bcdf314ef8b8c6d89fb9e2150a, State: Running, Role: LEADER
I20260928 02:34:35.132174 2665 consensus_queue.cc:237] T 647f4453ea154a6f9cecefeb1e5dc39d P 2cbf48bcdf314ef8b8c6d89fb9e2150a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:35.132690 2123 catalog_manager.cc:5495] T 647f4453ea154a6f9cecefeb1e5dc39d P 2cbf48bcdf314ef8b8c6d89fb9e2150a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cbf48bcdf314ef8b8c6d89fb9e2150a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } health_report { overall_health: UNKNOWN } } }
W20260928 02:34:35.155591 2607 tablet.cc:2369] T 647f4453ea154a6f9cecefeb1e5dc39d P d92c8bf86a9a47bfa8ea1e51855f3b0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:34:35.189895 2560 raft_consensus.cc:1261] T 647f4453ea154a6f9cecefeb1e5dc39d P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 1 FOLLOWER]: Refusing update from remote peer 2cbf48bcdf314ef8b8c6d89fb9e2150a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:35.189944 2420 raft_consensus.cc:1261] T 647f4453ea154a6f9cecefeb1e5dc39d P 249e2c930f94405f9f41c1e861b60c53 [term 1 FOLLOWER]: Refusing update from remote peer 2cbf48bcdf314ef8b8c6d89fb9e2150a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:35.190133 2636 consensus_queue.cc:1035] T 647f4453ea154a6f9cecefeb1e5dc39d P 2cbf48bcdf314ef8b8c6d89fb9e2150a [LEADER]: Connected to new peer: Peer: permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:35.190244 2665 consensus_queue.cc:1035] T 647f4453ea154a6f9cecefeb1e5dc39d P 2cbf48bcdf314ef8b8c6d89fb9e2150a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:34:35.194059 2651 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 02:34:36.709227 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:34:36.709337 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:34:36.710042 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d1911e566f74fc9b0af803869790bcc: Sending DeleteTablet for 3 replicas of tablet 647f4453ea154a6f9cecefeb1e5dc39d
I20260928 02:34:36.710275 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 647f4453ea154a6f9cecefeb1e5dc39d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37686
I20260928 02:34:36.710311 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 647f4453ea154a6f9cecefeb1e5dc39d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36566
I20260928 02:34:36.710326 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet 647f4453ea154a6f9cecefeb1e5dc39d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52382
I20260928 02:34:36.710551 2696 tablet_replica.cc:330] T 647f4453ea154a6f9cecefeb1e5dc39d P 249e2c930f94405f9f41c1e861b60c53: stopping tablet replica
I20260928 02:34:36.710651 2695 tablet_replica.cc:330] T 647f4453ea154a6f9cecefeb1e5dc39d P 2cbf48bcdf314ef8b8c6d89fb9e2150a: stopping tablet replica
I20260928 02:34:36.710664 2696 raft_consensus.cc:2229] T 647f4453ea154a6f9cecefeb1e5dc39d P 249e2c930f94405f9f41c1e861b60c53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:34:36.710701 2696 raft_consensus.cc:2258] T 647f4453ea154a6f9cecefeb1e5dc39d P 249e2c930f94405f9f41c1e861b60c53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:36.710764 2697 tablet_replica.cc:330] T 647f4453ea154a6f9cecefeb1e5dc39d P d92c8bf86a9a47bfa8ea1e51855f3b0a: stopping tablet replica
I20260928 02:34:36.710834 2697 raft_consensus.cc:2229] T 647f4453ea154a6f9cecefeb1e5dc39d P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:34:36.710873 2697 raft_consensus.cc:2258] T 647f4453ea154a6f9cecefeb1e5dc39d P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:36.710937 2695 raft_consensus.cc:2229] T 647f4453ea154a6f9cecefeb1e5dc39d P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:34:36.711001 2695 raft_consensus.cc:2258] T 647f4453ea154a6f9cecefeb1e5dc39d P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:36.711328 2697 ts_tablet_manager.cc:1905] T 647f4453ea154a6f9cecefeb1e5dc39d P d92c8bf86a9a47bfa8ea1e51855f3b0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:36.711326 2695 ts_tablet_manager.cc:1905] T 647f4453ea154a6f9cecefeb1e5dc39d P 2cbf48bcdf314ef8b8c6d89fb9e2150a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:36.713129 2695 ts_tablet_manager.cc:1918] T 647f4453ea154a6f9cecefeb1e5dc39d P 2cbf48bcdf314ef8b8c6d89fb9e2150a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:34:36.713212 2695 log.cc:1196] T 647f4453ea154a6f9cecefeb1e5dc39d P 2cbf48bcdf314ef8b8c6d89fb9e2150a: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-0/wal/wals/647f4453ea154a6f9cecefeb1e5dc39d
I20260928 02:34:36.713409 2696 ts_tablet_manager.cc:1905] T 647f4453ea154a6f9cecefeb1e5dc39d P 249e2c930f94405f9f41c1e861b60c53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:36.713483 2695 ts_tablet_manager.cc:1939] T 647f4453ea154a6f9cecefeb1e5dc39d P 2cbf48bcdf314ef8b8c6d89fb9e2150a: Deleting consensus metadata
I20260928 02:34:36.713774 2108 catalog_manager.cc:4841] TS 2cbf48bcdf314ef8b8c6d89fb9e2150a (127.0.98.1:42235): tablet 647f4453ea154a6f9cecefeb1e5dc39d (table test-table [id=2f53679cd36b4218b701d3c16c93321e]) successfully deleted
I20260928 02:34:36.714682 2697 ts_tablet_manager.cc:1918] T 647f4453ea154a6f9cecefeb1e5dc39d P d92c8bf86a9a47bfa8ea1e51855f3b0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:34:36.714727 2697 log.cc:1196] T 647f4453ea154a6f9cecefeb1e5dc39d P d92c8bf86a9a47bfa8ea1e51855f3b0a: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-2/wal/wals/647f4453ea154a6f9cecefeb1e5dc39d
I20260928 02:34:36.714747 2696 ts_tablet_manager.cc:1918] T 647f4453ea154a6f9cecefeb1e5dc39d P 249e2c930f94405f9f41c1e861b60c53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:34:36.714804 2696 log.cc:1196] T 647f4453ea154a6f9cecefeb1e5dc39d P 249e2c930f94405f9f41c1e861b60c53: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-1/wal/wals/647f4453ea154a6f9cecefeb1e5dc39d
I20260928 02:34:36.714915 2697 ts_tablet_manager.cc:1939] T 647f4453ea154a6f9cecefeb1e5dc39d P d92c8bf86a9a47bfa8ea1e51855f3b0a: Deleting consensus metadata
I20260928 02:34:36.715044 2696 ts_tablet_manager.cc:1939] T 647f4453ea154a6f9cecefeb1e5dc39d P 249e2c930f94405f9f41c1e861b60c53: Deleting consensus metadata
I20260928 02:34:36.715236 2110 catalog_manager.cc:4841] TS d92c8bf86a9a47bfa8ea1e51855f3b0a (127.0.98.3:36225): tablet 647f4453ea154a6f9cecefeb1e5dc39d (table test-table [id=2f53679cd36b4218b701d3c16c93321e]) successfully deleted
I20260928 02:34:36.715359 2108 catalog_manager.cc:4841] TS 249e2c930f94405f9f41c1e861b60c53 (127.0.98.2:40965): tablet 647f4453ea154a6f9cecefeb1e5dc39d (table test-table [id=2f53679cd36b4218b701d3c16c93321e]) successfully deleted
Sep 28 02:34:36 dist-test-slave-fz47 krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562876, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:34:36 dist-test-slave-fz47 krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562876, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:36.739887 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:36.728214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48572 (local address 127.0.98.62:44123)
0928 02:34:36.728330 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:36.728332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:36.728341 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:34:36.728389 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:36.728390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:36.728399 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:36.728455 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:36.728972 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:36.729507 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:36.730220 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:36.730417 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:36.730474 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:36.737768 (+ 7294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:36.737769 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:36.737770 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:36.737776 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:36.738930 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:36.739225 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:36.739226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:36.739226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:36.739287 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:36.739464 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:36.739465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:36.739465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:36.739580 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:36.739652 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:36.739744 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 02:34:36.739809 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":70,"threads_started":1}
I20260928 02:34:36.741415 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:34:36.741590 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:34:36.742722 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 36732260d4db494abc78b4b8b72d8fad (DEFAULT_TABLE table=test-table [id=ff6277f76ec7437aa1012e21f619d995]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:36.742743 2400 tablet_service.cc:1467] Processing CreateTablet for tablet 36732260d4db494abc78b4b8b72d8fad (DEFAULT_TABLE table=test-table [id=ff6277f76ec7437aa1012e21f619d995]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:36.742796 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 36732260d4db494abc78b4b8b72d8fad (DEFAULT_TABLE table=test-table [id=ff6277f76ec7437aa1012e21f619d995]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:36.742847 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36732260d4db494abc78b4b8b72d8fad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:36.742847 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36732260d4db494abc78b4b8b72d8fad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:36.742916 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36732260d4db494abc78b4b8b72d8fad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:36.743814 2706 tablet_bootstrap.cc:492] T 36732260d4db494abc78b4b8b72d8fad P d92c8bf86a9a47bfa8ea1e51855f3b0a: Bootstrap starting.
I20260928 02:34:36.744213 2706 tablet_bootstrap.cc:654] T 36732260d4db494abc78b4b8b72d8fad P d92c8bf86a9a47bfa8ea1e51855f3b0a: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:36.744359 2707 tablet_bootstrap.cc:492] T 36732260d4db494abc78b4b8b72d8fad P 249e2c930f94405f9f41c1e861b60c53: Bootstrap starting.
I20260928 02:34:36.744513 2708 tablet_bootstrap.cc:492] T 36732260d4db494abc78b4b8b72d8fad P 2cbf48bcdf314ef8b8c6d89fb9e2150a: Bootstrap starting.
I20260928 02:34:36.744731 2707 tablet_bootstrap.cc:654] T 36732260d4db494abc78b4b8b72d8fad P 249e2c930f94405f9f41c1e861b60c53: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:36.744868 2708 tablet_bootstrap.cc:654] T 36732260d4db494abc78b4b8b72d8fad P 2cbf48bcdf314ef8b8c6d89fb9e2150a: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:36.744961 2706 tablet_bootstrap.cc:492] T 36732260d4db494abc78b4b8b72d8fad P d92c8bf86a9a47bfa8ea1e51855f3b0a: No bootstrap required, opened a new log
I20260928 02:34:36.745011 2706 ts_tablet_manager.cc:1397] T 36732260d4db494abc78b4b8b72d8fad P d92c8bf86a9a47bfa8ea1e51855f3b0a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:34:36.745134 2706 raft_consensus.cc:348] T 36732260d4db494abc78b4b8b72d8fad P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:36.745208 2706 raft_consensus.cc:374] T 36732260d4db494abc78b4b8b72d8fad P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:36.745226 2706 raft_consensus.cc:729] T 36732260d4db494abc78b4b8b72d8fad P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92c8bf86a9a47bfa8ea1e51855f3b0a, State: Initialized, Role: FOLLOWER
I20260928 02:34:36.745357 2707 tablet_bootstrap.cc:492] T 36732260d4db494abc78b4b8b72d8fad P 249e2c930f94405f9f41c1e861b60c53: No bootstrap required, opened a new log
I20260928 02:34:36.745337 2706 consensus_queue.cc:260] T 36732260d4db494abc78b4b8b72d8fad P d92c8bf86a9a47bfa8ea1e51855f3b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:36.745406 2707 ts_tablet_manager.cc:1397] T 36732260d4db494abc78b4b8b72d8fad P 249e2c930f94405f9f41c1e861b60c53: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:34:36.745493 2706 ts_tablet_manager.cc:1428] T 36732260d4db494abc78b4b8b72d8fad P d92c8bf86a9a47bfa8ea1e51855f3b0a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:36.745551 2707 raft_consensus.cc:348] T 36732260d4db494abc78b4b8b72d8fad P 249e2c930f94405f9f41c1e861b60c53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:36.745618 2707 raft_consensus.cc:374] T 36732260d4db494abc78b4b8b72d8fad P 249e2c930f94405f9f41c1e861b60c53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:36.745640 2707 raft_consensus.cc:729] T 36732260d4db494abc78b4b8b72d8fad P 249e2c930f94405f9f41c1e861b60c53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 249e2c930f94405f9f41c1e861b60c53, State: Initialized, Role: FOLLOWER
I20260928 02:34:36.745676 2707 consensus_queue.cc:260] T 36732260d4db494abc78b4b8b72d8fad P 249e2c930f94405f9f41c1e861b60c53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:36.745827 2707 ts_tablet_manager.cc:1428] T 36732260d4db494abc78b4b8b72d8fad P 249e2c930f94405f9f41c1e861b60c53: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:36.746026 2708 tablet_bootstrap.cc:492] T 36732260d4db494abc78b4b8b72d8fad P 2cbf48bcdf314ef8b8c6d89fb9e2150a: No bootstrap required, opened a new log
I20260928 02:34:36.746079 2708 ts_tablet_manager.cc:1397] T 36732260d4db494abc78b4b8b72d8fad P 2cbf48bcdf314ef8b8c6d89fb9e2150a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:34:36.746187 2708 raft_consensus.cc:348] T 36732260d4db494abc78b4b8b72d8fad P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:36.746259 2708 raft_consensus.cc:374] T 36732260d4db494abc78b4b8b72d8fad P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:36.746279 2708 raft_consensus.cc:729] T 36732260d4db494abc78b4b8b72d8fad P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cbf48bcdf314ef8b8c6d89fb9e2150a, State: Initialized, Role: FOLLOWER
I20260928 02:34:36.746317 2708 consensus_queue.cc:260] T 36732260d4db494abc78b4b8b72d8fad P 2cbf48bcdf314ef8b8c6d89fb9e2150a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:36.746416 2708 ts_tablet_manager.cc:1428] T 36732260d4db494abc78b4b8b72d8fad P 2cbf48bcdf314ef8b8c6d89fb9e2150a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:37.194444 2678 raft_consensus.cc:482] T 36732260d4db494abc78b4b8b72d8fad P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:34:37.194533 2678 raft_consensus.cc:504] T 36732260d4db494abc78b4b8b72d8fad P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:37.194687 2678 leader_election.cc:288] T 36732260d4db494abc78b4b8b72d8fad P 2cbf48bcdf314ef8b8c6d89fb9e2150a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 249e2c930f94405f9f41c1e861b60c53 (127.0.98.2:40965), d92c8bf86a9a47bfa8ea1e51855f3b0a (127.0.98.3:36225)
I20260928 02:34:37.194862 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36732260d4db494abc78b4b8b72d8fad" candidate_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "249e2c930f94405f9f41c1e861b60c53" is_pre_election: true
I20260928 02:34:37.194896 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36732260d4db494abc78b4b8b72d8fad" candidate_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" is_pre_election: true
I20260928 02:34:37.194968 2420 raft_consensus.cc:2454] T 36732260d4db494abc78b4b8b72d8fad P 249e2c930f94405f9f41c1e861b60c53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cbf48bcdf314ef8b8c6d89fb9e2150a in term 0.
I20260928 02:34:37.194979 2560 raft_consensus.cc:2454] T 36732260d4db494abc78b4b8b72d8fad P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cbf48bcdf314ef8b8c6d89fb9e2150a in term 0.
I20260928 02:34:37.195142 2212 leader_election.cc:302] T 36732260d4db494abc78b4b8b72d8fad P 2cbf48bcdf314ef8b8c6d89fb9e2150a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 249e2c930f94405f9f41c1e861b60c53, 2cbf48bcdf314ef8b8c6d89fb9e2150a; no voters:
I20260928 02:34:37.195255 2678 raft_consensus.cc:2781] T 36732260d4db494abc78b4b8b72d8fad P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:34:37.195307 2678 raft_consensus.cc:482] T 36732260d4db494abc78b4b8b72d8fad P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:34:37.195326 2678 raft_consensus.cc:3037] T 36732260d4db494abc78b4b8b72d8fad P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:37.195986 2678 raft_consensus.cc:504] T 36732260d4db494abc78b4b8b72d8fad P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:37.196105 2678 leader_election.cc:288] T 36732260d4db494abc78b4b8b72d8fad P 2cbf48bcdf314ef8b8c6d89fb9e2150a [CANDIDATE]: Term 1 election: Requested vote from peers 249e2c930f94405f9f41c1e861b60c53 (127.0.98.2:40965), d92c8bf86a9a47bfa8ea1e51855f3b0a (127.0.98.3:36225)
I20260928 02:34:37.196252 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36732260d4db494abc78b4b8b72d8fad" candidate_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a"
I20260928 02:34:37.196260 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36732260d4db494abc78b4b8b72d8fad" candidate_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "249e2c930f94405f9f41c1e861b60c53"
I20260928 02:34:37.196321 2420 raft_consensus.cc:3037] T 36732260d4db494abc78b4b8b72d8fad P 249e2c930f94405f9f41c1e861b60c53 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:37.196321 2560 raft_consensus.cc:3037] T 36732260d4db494abc78b4b8b72d8fad P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:37.196856 2420 raft_consensus.cc:2454] T 36732260d4db494abc78b4b8b72d8fad P 249e2c930f94405f9f41c1e861b60c53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cbf48bcdf314ef8b8c6d89fb9e2150a in term 1.
I20260928 02:34:37.196866 2560 raft_consensus.cc:2454] T 36732260d4db494abc78b4b8b72d8fad P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cbf48bcdf314ef8b8c6d89fb9e2150a in term 1.
I20260928 02:34:37.197007 2214 leader_election.cc:302] T 36732260d4db494abc78b4b8b72d8fad P 2cbf48bcdf314ef8b8c6d89fb9e2150a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cbf48bcdf314ef8b8c6d89fb9e2150a, d92c8bf86a9a47bfa8ea1e51855f3b0a; no voters:
I20260928 02:34:37.197101 2678 raft_consensus.cc:2781] T 36732260d4db494abc78b4b8b72d8fad P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:37.197160 2678 raft_consensus.cc:686] T 36732260d4db494abc78b4b8b72d8fad P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 1 LEADER]: Becoming Leader. State: Replica: 2cbf48bcdf314ef8b8c6d89fb9e2150a, State: Running, Role: LEADER
I20260928 02:34:37.197211 2678 consensus_queue.cc:237] T 36732260d4db494abc78b4b8b72d8fad P 2cbf48bcdf314ef8b8c6d89fb9e2150a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:37.197850 2123 catalog_manager.cc:5495] T 36732260d4db494abc78b4b8b72d8fad P 2cbf48bcdf314ef8b8c6d89fb9e2150a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cbf48bcdf314ef8b8c6d89fb9e2150a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } health_report { overall_health: UNKNOWN } } }
I20260928 02:34:37.295454 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:37.293410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37724 (local address 127.0.98.1:42235)
0928 02:34:37.293529 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:37.293531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:37.293540 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:34:37.293571 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:37.293572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:37.293581 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:37.293632 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:34:37.293721 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:37.294201 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:37.294894 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:37.295098 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:37.295150 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:37.295260 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:34:37.295286 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:34:37.295321 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:37.295357 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 02:34:37.295374 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":53,"threads_started":1}
I20260928 02:34:37.297340 2560 raft_consensus.cc:1261] T 36732260d4db494abc78b4b8b72d8fad P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 1 FOLLOWER]: Refusing update from remote peer 2cbf48bcdf314ef8b8c6d89fb9e2150a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:37.297377 2420 raft_consensus.cc:1261] T 36732260d4db494abc78b4b8b72d8fad P 249e2c930f94405f9f41c1e861b60c53 [term 1 FOLLOWER]: Refusing update from remote peer 2cbf48bcdf314ef8b8c6d89fb9e2150a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:37.297557 2718 consensus_queue.cc:1035] T 36732260d4db494abc78b4b8b72d8fad P 2cbf48bcdf314ef8b8c6d89fb9e2150a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:37.297665 2636 consensus_queue.cc:1035] T 36732260d4db494abc78b4b8b72d8fad P 2cbf48bcdf314ef8b8c6d89fb9e2150a [LEADER]: Connected to new peer: Peer: permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:37.306404 2123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:34:37.306488 2123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:34:37.307010 2123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d1911e566f74fc9b0af803869790bcc: Sending DeleteTablet for 3 replicas of tablet 36732260d4db494abc78b4b8b72d8fad
I20260928 02:34:37.307214 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 36732260d4db494abc78b4b8b72d8fad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37686
I20260928 02:34:37.307245 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet 36732260d4db494abc78b4b8b72d8fad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52382
I20260928 02:34:37.307262 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 36732260d4db494abc78b4b8b72d8fad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36566
I20260928 02:34:37.307417 2733 tablet_replica.cc:330] T 36732260d4db494abc78b4b8b72d8fad P d92c8bf86a9a47bfa8ea1e51855f3b0a: stopping tablet replica
I20260928 02:34:37.307466 2733 raft_consensus.cc:2229] T 36732260d4db494abc78b4b8b72d8fad P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:34:37.307510 2733 raft_consensus.cc:2258] T 36732260d4db494abc78b4b8b72d8fad P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:37.307559 2732 tablet_replica.cc:330] T 36732260d4db494abc78b4b8b72d8fad P 249e2c930f94405f9f41c1e861b60c53: stopping tablet replica
I20260928 02:34:37.307616 2732 raft_consensus.cc:2229] T 36732260d4db494abc78b4b8b72d8fad P 249e2c930f94405f9f41c1e861b60c53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:34:37.307651 2732 raft_consensus.cc:2258] T 36732260d4db494abc78b4b8b72d8fad P 249e2c930f94405f9f41c1e861b60c53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:37.307598 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:34:37.307734 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:34:37.307891 2732 ts_tablet_manager.cc:1905] T 36732260d4db494abc78b4b8b72d8fad P 249e2c930f94405f9f41c1e861b60c53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:37.308039 2733 ts_tablet_manager.cc:1905] T 36732260d4db494abc78b4b8b72d8fad P d92c8bf86a9a47bfa8ea1e51855f3b0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:37.308843 2259 tablet_service.cc:1467] Processing CreateTablet for tablet c721c7202008424ea90b0d6251e90d6b (DEFAULT_TABLE table=test-table [id=bd558947875044298b3b975e8adccd50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:37.308957 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c721c7202008424ea90b0d6251e90d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:37.309343 2731 tablet_replica.cc:330] T 36732260d4db494abc78b4b8b72d8fad P 2cbf48bcdf314ef8b8c6d89fb9e2150a: stopping tablet replica
I20260928 02:34:37.309412 2731 raft_consensus.cc:2229] T 36732260d4db494abc78b4b8b72d8fad P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:34:37.309465 2731 raft_consensus.cc:2258] T 36732260d4db494abc78b4b8b72d8fad P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:37.309580 2732 ts_tablet_manager.cc:1918] T 36732260d4db494abc78b4b8b72d8fad P 249e2c930f94405f9f41c1e861b60c53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:34:37.309602 2733 ts_tablet_manager.cc:1918] T 36732260d4db494abc78b4b8b72d8fad P d92c8bf86a9a47bfa8ea1e51855f3b0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:34:37.309648 2731 ts_tablet_manager.cc:1905] T 36732260d4db494abc78b4b8b72d8fad P 2cbf48bcdf314ef8b8c6d89fb9e2150a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:37.309662 2733 log.cc:1196] T 36732260d4db494abc78b4b8b72d8fad P d92c8bf86a9a47bfa8ea1e51855f3b0a: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-2/wal/wals/36732260d4db494abc78b4b8b72d8fad
I20260928 02:34:37.309741 2732 log.cc:1196] T 36732260d4db494abc78b4b8b72d8fad P 249e2c930f94405f9f41c1e861b60c53: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-1/wal/wals/36732260d4db494abc78b4b8b72d8fad
I20260928 02:34:37.309898 2733 ts_tablet_manager.cc:1939] T 36732260d4db494abc78b4b8b72d8fad P d92c8bf86a9a47bfa8ea1e51855f3b0a: Deleting consensus metadata
I20260928 02:34:37.310072 2732 ts_tablet_manager.cc:1939] T 36732260d4db494abc78b4b8b72d8fad P 249e2c930f94405f9f41c1e861b60c53: Deleting consensus metadata
I20260928 02:34:37.310160 2110 catalog_manager.cc:4841] TS d92c8bf86a9a47bfa8ea1e51855f3b0a (127.0.98.3:36225): tablet 36732260d4db494abc78b4b8b72d8fad (table test-table [id=ff6277f76ec7437aa1012e21f619d995]) successfully deleted
I20260928 02:34:37.310407 2400 tablet_service.cc:1467] Processing CreateTablet for tablet c721c7202008424ea90b0d6251e90d6b (DEFAULT_TABLE table=test-table [id=bd558947875044298b3b975e8adccd50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:37.310442 2734 tablet_bootstrap.cc:492] T c721c7202008424ea90b0d6251e90d6b P 2cbf48bcdf314ef8b8c6d89fb9e2150a: Bootstrap starting.
I20260928 02:34:37.310503 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c721c7202008424ea90b0d6251e90d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:37.310729 2734 tablet_bootstrap.cc:654] T c721c7202008424ea90b0d6251e90d6b P 2cbf48bcdf314ef8b8c6d89fb9e2150a: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:37.310986 2731 ts_tablet_manager.cc:1918] T 36732260d4db494abc78b4b8b72d8fad P 2cbf48bcdf314ef8b8c6d89fb9e2150a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:34:37.311049 2731 log.cc:1196] T 36732260d4db494abc78b4b8b72d8fad P 2cbf48bcdf314ef8b8c6d89fb9e2150a: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-0/wal/wals/36732260d4db494abc78b4b8b72d8fad
I20260928 02:34:37.311233 2731 ts_tablet_manager.cc:1939] T 36732260d4db494abc78b4b8b72d8fad P 2cbf48bcdf314ef8b8c6d89fb9e2150a: Deleting consensus metadata
I20260928 02:34:37.311236 2734 tablet_bootstrap.cc:492] T c721c7202008424ea90b0d6251e90d6b P 2cbf48bcdf314ef8b8c6d89fb9e2150a: No bootstrap required, opened a new log
I20260928 02:34:37.311292 2734 ts_tablet_manager.cc:1397] T c721c7202008424ea90b0d6251e90d6b P 2cbf48bcdf314ef8b8c6d89fb9e2150a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:34:37.311386 2734 raft_consensus.cc:348] T c721c7202008424ea90b0d6251e90d6b P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:37.311443 2734 raft_consensus.cc:374] T c721c7202008424ea90b0d6251e90d6b P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:37.311455 2734 raft_consensus.cc:729] T c721c7202008424ea90b0d6251e90d6b P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cbf48bcdf314ef8b8c6d89fb9e2150a, State: Initialized, Role: FOLLOWER
I20260928 02:34:37.311511 2734 consensus_queue.cc:260] T c721c7202008424ea90b0d6251e90d6b P 2cbf48bcdf314ef8b8c6d89fb9e2150a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:37.311693 2734 ts_tablet_manager.cc:1428] T c721c7202008424ea90b0d6251e90d6b P 2cbf48bcdf314ef8b8c6d89fb9e2150a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:37.311702 2108 catalog_manager.cc:4841] TS 2cbf48bcdf314ef8b8c6d89fb9e2150a (127.0.98.1:42235): tablet 36732260d4db494abc78b4b8b72d8fad (table test-table [id=ff6277f76ec7437aa1012e21f619d995]) successfully deleted
I20260928 02:34:37.311751 2108 catalog_manager.cc:4841] TS 249e2c930f94405f9f41c1e861b60c53 (127.0.98.2:40965): tablet 36732260d4db494abc78b4b8b72d8fad (table test-table [id=ff6277f76ec7437aa1012e21f619d995]) successfully deleted
I20260928 02:34:37.311901 2540 tablet_service.cc:1467] Processing CreateTablet for tablet c721c7202008424ea90b0d6251e90d6b (DEFAULT_TABLE table=test-table [id=bd558947875044298b3b975e8adccd50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:37.312011 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c721c7202008424ea90b0d6251e90d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:37.312776 2737 tablet_bootstrap.cc:492] T c721c7202008424ea90b0d6251e90d6b P d92c8bf86a9a47bfa8ea1e51855f3b0a: Bootstrap starting.
I20260928 02:34:37.312906 2736 tablet_bootstrap.cc:492] T c721c7202008424ea90b0d6251e90d6b P 249e2c930f94405f9f41c1e861b60c53: Bootstrap starting.
I20260928 02:34:37.313225 2737 tablet_bootstrap.cc:654] T c721c7202008424ea90b0d6251e90d6b P d92c8bf86a9a47bfa8ea1e51855f3b0a: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:37.313235 2736 tablet_bootstrap.cc:654] T c721c7202008424ea90b0d6251e90d6b P 249e2c930f94405f9f41c1e861b60c53: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:37.313748 2736 tablet_bootstrap.cc:492] T c721c7202008424ea90b0d6251e90d6b P 249e2c930f94405f9f41c1e861b60c53: No bootstrap required, opened a new log
I20260928 02:34:37.313792 2736 ts_tablet_manager.cc:1397] T c721c7202008424ea90b0d6251e90d6b P 249e2c930f94405f9f41c1e861b60c53: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:34:37.313830 2737 tablet_bootstrap.cc:492] T c721c7202008424ea90b0d6251e90d6b P d92c8bf86a9a47bfa8ea1e51855f3b0a: No bootstrap required, opened a new log
I20260928 02:34:37.313874 2737 ts_tablet_manager.cc:1397] T c721c7202008424ea90b0d6251e90d6b P d92c8bf86a9a47bfa8ea1e51855f3b0a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 02:34:37.313915 2736 raft_consensus.cc:348] T c721c7202008424ea90b0d6251e90d6b P 249e2c930f94405f9f41c1e861b60c53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:37.313988 2736 raft_consensus.cc:374] T c721c7202008424ea90b0d6251e90d6b P 249e2c930f94405f9f41c1e861b60c53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:37.314002 2736 raft_consensus.cc:729] T c721c7202008424ea90b0d6251e90d6b P 249e2c930f94405f9f41c1e861b60c53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 249e2c930f94405f9f41c1e861b60c53, State: Initialized, Role: FOLLOWER
I20260928 02:34:37.314000 2737 raft_consensus.cc:348] T c721c7202008424ea90b0d6251e90d6b P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:37.314033 2736 consensus_queue.cc:260] T c721c7202008424ea90b0d6251e90d6b P 249e2c930f94405f9f41c1e861b60c53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:37.314061 2737 raft_consensus.cc:374] T c721c7202008424ea90b0d6251e90d6b P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:37.314082 2737 raft_consensus.cc:729] T c721c7202008424ea90b0d6251e90d6b P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92c8bf86a9a47bfa8ea1e51855f3b0a, State: Initialized, Role: FOLLOWER
I20260928 02:34:37.314109 2736 ts_tablet_manager.cc:1428] T c721c7202008424ea90b0d6251e90d6b P 249e2c930f94405f9f41c1e861b60c53: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:37.314134 2737 consensus_queue.cc:260] T c721c7202008424ea90b0d6251e90d6b P d92c8bf86a9a47bfa8ea1e51855f3b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:37.314218 2737 ts_tablet_manager.cc:1428] T c721c7202008424ea90b0d6251e90d6b P d92c8bf86a9a47bfa8ea1e51855f3b0a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:37.449149 2726 raft_consensus.cc:482] T c721c7202008424ea90b0d6251e90d6b P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:34:37.449236 2726 raft_consensus.cc:504] T c721c7202008424ea90b0d6251e90d6b P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:37.449559 2726 leader_election.cc:288] T c721c7202008424ea90b0d6251e90d6b P d92c8bf86a9a47bfa8ea1e51855f3b0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cbf48bcdf314ef8b8c6d89fb9e2150a (127.0.98.1:42235), 249e2c930f94405f9f41c1e861b60c53 (127.0.98.2:40965)
I20260928 02:34:37.452567 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:37.449723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42235 (local address 127.0.98.3:55059)
0928 02:34:37.449881 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:37.449891 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:37.449964 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:37.450145 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:37.450146 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:37.450148 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:37.450223 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:37.450225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:37.450852 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:37.450852 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:37.451832 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:37.451834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:37.452379 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:37.452379 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:37.452435 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:37.452439 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:34:37.452474 (+ 35us) client_negotiation.cc:240] Negotiation successful
0928 02:34:37.452505 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":94,"threads_started":1}
I20260928 02:34:37.452590 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:37.449782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55059 (local address 127.0.98.1:42235)
0928 02:34:37.449849 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:37.449851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:37.449965 (+ 114us) server_negotiation.cc:407] Connection header received
0928 02:34:37.450067 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:37.450068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:37.450081 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:37.450149 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:37.450293 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:37.450778 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:37.451914 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:37.452307 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:37.452375 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:37.452385 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:37.452482 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 02:34:37.452514 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:34:37.452786 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c721c7202008424ea90b0d6251e90d6b" candidate_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" is_pre_election: true
I20260928 02:34:37.452914 2280 raft_consensus.cc:2454] T c721c7202008424ea90b0d6251e90d6b P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d92c8bf86a9a47bfa8ea1e51855f3b0a in term 0.
I20260928 02:34:37.453110 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:37.449919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40965 (local address 127.0.98.3:59881)
0928 02:34:37.450019 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:37.450028 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:37.450078 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:37.450279 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:37.450279 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:37.450281 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:37.450721 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:37.450722 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:37.451361 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:37.451362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:37.452228 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:37.452230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:37.452879 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:37.452880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:37.452948 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:37.452951 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:34:37.453002 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 02:34:37.453036 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":68,"threads_started":1}
I20260928 02:34:37.453109 2492 leader_election.cc:302] T c721c7202008424ea90b0d6251e90d6b P d92c8bf86a9a47bfa8ea1e51855f3b0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cbf48bcdf314ef8b8c6d89fb9e2150a, d92c8bf86a9a47bfa8ea1e51855f3b0a; no voters:
I20260928 02:34:37.453212 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:37.450003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59881 (local address 127.0.98.2:40965)
0928 02:34:37.450165 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:37.450167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:37.450174 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:34:37.450199 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:37.450200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:37.450207 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:37.450263 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:37.450789 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:37.451271 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:37.452299 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:37.452803 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:37.452870 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:37.452882 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:37.453106 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 02:34:37.453142 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
I20260928 02:34:37.453233 2726 raft_consensus.cc:2781] T c721c7202008424ea90b0d6251e90d6b P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:34:37.453275 2726 raft_consensus.cc:482] T c721c7202008424ea90b0d6251e90d6b P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:34:37.453295 2726 raft_consensus.cc:3037] T c721c7202008424ea90b0d6251e90d6b P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:37.453414 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c721c7202008424ea90b0d6251e90d6b" candidate_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "249e2c930f94405f9f41c1e861b60c53" is_pre_election: true
I20260928 02:34:37.453490 2420 raft_consensus.cc:2454] T c721c7202008424ea90b0d6251e90d6b P 249e2c930f94405f9f41c1e861b60c53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d92c8bf86a9a47bfa8ea1e51855f3b0a in term 0.
I20260928 02:34:37.454080 2726 raft_consensus.cc:504] T c721c7202008424ea90b0d6251e90d6b P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:37.454195 2726 leader_election.cc:288] T c721c7202008424ea90b0d6251e90d6b P d92c8bf86a9a47bfa8ea1e51855f3b0a [CANDIDATE]: Term 1 election: Requested vote from peers 2cbf48bcdf314ef8b8c6d89fb9e2150a (127.0.98.1:42235), 249e2c930f94405f9f41c1e861b60c53 (127.0.98.2:40965)
I20260928 02:34:37.454341 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c721c7202008424ea90b0d6251e90d6b" candidate_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a"
I20260928 02:34:37.454350 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c721c7202008424ea90b0d6251e90d6b" candidate_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "249e2c930f94405f9f41c1e861b60c53"
I20260928 02:34:37.454401 2280 raft_consensus.cc:3037] T c721c7202008424ea90b0d6251e90d6b P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:37.454411 2420 raft_consensus.cc:3037] T c721c7202008424ea90b0d6251e90d6b P 249e2c930f94405f9f41c1e861b60c53 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:37.454928 2420 raft_consensus.cc:2454] T c721c7202008424ea90b0d6251e90d6b P 249e2c930f94405f9f41c1e861b60c53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d92c8bf86a9a47bfa8ea1e51855f3b0a in term 1.
I20260928 02:34:37.455000 2280 raft_consensus.cc:2454] T c721c7202008424ea90b0d6251e90d6b P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d92c8bf86a9a47bfa8ea1e51855f3b0a in term 1.
I20260928 02:34:37.455066 2492 leader_election.cc:302] T c721c7202008424ea90b0d6251e90d6b P d92c8bf86a9a47bfa8ea1e51855f3b0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 249e2c930f94405f9f41c1e861b60c53, d92c8bf86a9a47bfa8ea1e51855f3b0a; no voters:
I20260928 02:34:37.455152 2726 raft_consensus.cc:2781] T c721c7202008424ea90b0d6251e90d6b P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:37.455268 2726 raft_consensus.cc:686] T c721c7202008424ea90b0d6251e90d6b P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 1 LEADER]: Becoming Leader. State: Replica: d92c8bf86a9a47bfa8ea1e51855f3b0a, State: Running, Role: LEADER
I20260928 02:34:37.455324 2726 consensus_queue.cc:237] T c721c7202008424ea90b0d6251e90d6b P d92c8bf86a9a47bfa8ea1e51855f3b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } }
I20260928 02:34:37.455911 2123 catalog_manager.cc:5495] T c721c7202008424ea90b0d6251e90d6b P d92c8bf86a9a47bfa8ea1e51855f3b0a reported cstate change: term changed from 0 to 1, leader changed from <none> to d92c8bf86a9a47bfa8ea1e51855f3b0a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d92c8bf86a9a47bfa8ea1e51855f3b0a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36225 } health_report { overall_health: HEALTHY } } }
I20260928 02:34:37.494822 2745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:37.492682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36580 (local address 127.0.98.3:36225)
0928 02:34:37.492817 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:37.492819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:37.492823 (+ 4us) server_negotiation.cc:407] Connection header received
0928 02:34:37.492853 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:37.492853 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:37.492871 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:37.492928 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:34:37.493038 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:37.493584 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:37.494238 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:37.494421 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:37.494464 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:37.494551 (+ 87us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:34:37.494650 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:34:37.494698 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:37.494712 (+ 14us) server_negotiation.cc:299] Negotiation successful
0928 02:34:37.494726 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":92,"threads_started":1}
I20260928 02:34:37.499115 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:37.496969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48337 (local address 127.0.98.1:42235)
0928 02:34:37.497025 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:37.497027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:37.497069 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:34:37.497146 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:37.497147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:37.497158 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:37.497197 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:34:37.497333 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:37.497785 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:37.498509 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:37.498698 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:37.498748 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:37.498902 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:34:37.498931 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:34:37.498970 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:37.499023 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 02:34:37.499044 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:34:37.499161 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:37.496894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42235 (local address 127.0.98.3:48337)
0928 02:34:37.497012 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:37.497022 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:37.497088 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:37.497207 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:37.497208 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:37.497209 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:34:37.497284 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:37.497285 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:37.497855 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:37.497856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:37.498430 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:37.498432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:37.498758 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:37.498759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:37.498821 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:37.498832 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:34:37.498979 (+ 147us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:34:37.498981 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:34:37.499020 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 02:34:37.499059 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":58,"threads_started":1}
I20260928 02:34:37.500936 2420 raft_consensus.cc:1261] T c721c7202008424ea90b0d6251e90d6b P 249e2c930f94405f9f41c1e861b60c53 [term 1 FOLLOWER]: Refusing update from remote peer d92c8bf86a9a47bfa8ea1e51855f3b0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:37.501024 2280 raft_consensus.cc:1261] T c721c7202008424ea90b0d6251e90d6b P 2cbf48bcdf314ef8b8c6d89fb9e2150a [term 1 FOLLOWER]: Refusing update from remote peer d92c8bf86a9a47bfa8ea1e51855f3b0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:37.501217 2726 consensus_queue.cc:1035] T c721c7202008424ea90b0d6251e90d6b P d92c8bf86a9a47bfa8ea1e51855f3b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "249e2c930f94405f9f41c1e861b60c53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:37.501353 2743 consensus_queue.cc:1035] T c721c7202008424ea90b0d6251e90d6b P d92c8bf86a9a47bfa8ea1e51855f3b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cbf48bcdf314ef8b8c6d89fb9e2150a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:37.508111 2745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:37.505663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40289 (local address 127.0.98.3:36225)
0928 02:34:37.505696 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:37.505698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:37.505740 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:34:37.505826 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:37.505826 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:37.505835 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:37.505861 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:34:37.506021 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:37.506712 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:37.507596 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:37.507798 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:37.507845 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:37.507938 (+ 93us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:34:37.507966 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:34:37.508005 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:37.508037 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 02:34:37.508051 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 02:34:37.508116 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:37.505574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36225 (local address 127.0.98.1:40289)
0928 02:34:37.505684 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:37.505693 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:37.505762 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:37.505915 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:37.505915 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:37.505916 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:34:37.505956 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:37.505957 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:37.506779 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:37.506779 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:37.507521 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:37.507523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:37.507850 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:37.507850 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:37.507885 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:37.507889 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:34:37.508006 (+ 117us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:34:37.508007 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 02:34:37.508035 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 02:34:37.508063 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":72,"threads_started":1}
W20260928 02:34:37.508433 2717 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 02:34:39.030071 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:34:39.030167 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:34:39.030838 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d1911e566f74fc9b0af803869790bcc: Sending DeleteTablet for 3 replicas of tablet c721c7202008424ea90b0d6251e90d6b
I20260928 02:34:39.031137 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet c721c7202008424ea90b0d6251e90d6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37686
I20260928 02:34:39.031150 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet c721c7202008424ea90b0d6251e90d6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52382
I20260928 02:34:39.031172 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet c721c7202008424ea90b0d6251e90d6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36566
I20260928 02:34:39.031323 2778 tablet_replica.cc:330] T c721c7202008424ea90b0d6251e90d6b P d92c8bf86a9a47bfa8ea1e51855f3b0a: stopping tablet replica
I20260928 02:34:39.031340 2779 tablet_replica.cc:330] T c721c7202008424ea90b0d6251e90d6b P 249e2c930f94405f9f41c1e861b60c53: stopping tablet replica
I20260928 02:34:39.031381 2779 raft_consensus.cc:2229] T c721c7202008424ea90b0d6251e90d6b P 249e2c930f94405f9f41c1e861b60c53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:34:39.031404 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 2195
I20260928 02:34:39.031428 2779 raft_consensus.cc:2258] T c721c7202008424ea90b0d6251e90d6b P 249e2c930f94405f9f41c1e861b60c53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:39.031458 2778 raft_consensus.cc:2229] T c721c7202008424ea90b0d6251e90d6b P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:34:39.032692 2778 raft_consensus.cc:2258] T c721c7202008424ea90b0d6251e90d6b P d92c8bf86a9a47bfa8ea1e51855f3b0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:39.032945 2778 ts_tablet_manager.cc:1905] T c721c7202008424ea90b0d6251e90d6b P d92c8bf86a9a47bfa8ea1e51855f3b0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:39.033507 2779 ts_tablet_manager.cc:1905] T c721c7202008424ea90b0d6251e90d6b P 249e2c930f94405f9f41c1e861b60c53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:39.034680 2778 ts_tablet_manager.cc:1918] T c721c7202008424ea90b0d6251e90d6b P d92c8bf86a9a47bfa8ea1e51855f3b0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:34:39.034739 2778 log.cc:1196] T c721c7202008424ea90b0d6251e90d6b P d92c8bf86a9a47bfa8ea1e51855f3b0a: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-2/wal/wals/c721c7202008424ea90b0d6251e90d6b
I20260928 02:34:39.034937 2778 ts_tablet_manager.cc:1939] T c721c7202008424ea90b0d6251e90d6b P d92c8bf86a9a47bfa8ea1e51855f3b0a: Deleting consensus metadata
I20260928 02:34:39.034976 2779 ts_tablet_manager.cc:1918] T c721c7202008424ea90b0d6251e90d6b P 249e2c930f94405f9f41c1e861b60c53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:34:39.035073 2779 log.cc:1196] T c721c7202008424ea90b0d6251e90d6b P 249e2c930f94405f9f41c1e861b60c53: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790562862804661-392-0/minicluster-data/ts-1/wal/wals/c721c7202008424ea90b0d6251e90d6b
I20260928 02:34:39.035225 2110 catalog_manager.cc:4841] TS d92c8bf86a9a47bfa8ea1e51855f3b0a (127.0.98.3:36225): tablet c721c7202008424ea90b0d6251e90d6b (table test-table [id=bd558947875044298b3b975e8adccd50]) successfully deleted
I20260928 02:34:39.035310 2779 ts_tablet_manager.cc:1939] T c721c7202008424ea90b0d6251e90d6b P 249e2c930f94405f9f41c1e861b60c53: Deleting consensus metadata
I20260928 02:34:39.035552 2108 catalog_manager.cc:4841] TS 249e2c930f94405f9f41c1e861b60c53 (127.0.98.2:40965): tablet c721c7202008424ea90b0d6251e90d6b (table test-table [id=bd558947875044298b3b975e8adccd50]) successfully deleted
W20260928 02:34:39.036795 2108 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42235 (error 108)
I20260928 02:34:39.037041 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 2335
I20260928 02:34:39.040762 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 2475
I20260928 02:34:39.040835 2781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:39.037099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42235 (local address 127.0.0.1:37738)
0928 02:34:39.040756 (+ 3657us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:39.040796 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3637,"thread_start_us":55,"threads_started":1}
W20260928 02:34:39.041396 2108 catalog_manager.cc:4584] TS 2cbf48bcdf314ef8b8c6d89fb9e2150a (127.0.98.1:42235): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c721c7202008424ea90b0d6251e90d6b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42235: connect: Connection refused (error 111)
I20260928 02:34:39.045846 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 2091
2026-09-28T02:34:39Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6054 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:34:39 dist-test-slave-fz47 krb5kdc[2784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:34:39 dist-test-slave-fz47 krb5kdc[2784](info): set up 2 sockets
Sep 28 02:34:39 dist-test-slave-fz47 krb5kdc[2784](info): commencing operation
krb5kdc: starting...
W20260928 02:34:39.988837 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:34:40 dist-test-slave-fz47 krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:34:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:34:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 02:34:40.095742 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43141
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:40.172672 2797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:40.172992 2797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:40.173062 2797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:40.174284 2797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:34:40.174340 2797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:40.174358 2797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:40.174374 2797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:34:40.174389 2797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:34:40.175887 2797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:40.176122 2797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:40.176333 2797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:34:40.181177 2797 server_base.cc:1034] running on GCE node
W20260928 02:34:40.181169 2802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:40.181234 2805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:40.181376 2803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:40.181624 2797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:40.181877 2797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:40.183017 2797 hybrid_clock.cc:648] HybridClock initialized: now 1790562880182996 us; error 32 us; skew 500 ppm
Sep 28 02:34:40 dist-test-slave-fz47 krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:40.185158 2797 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:34:40.185607 2797 webserver.cc:460] Webserver started at http://127.0.98.62:46883/ using document root <none> and password file <none>
I20260928 02:34:40.185807 2797 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:40.185853 2797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:40.185968 2797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:40.186947 2797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "c6d2f7123bf44741a51663c40e351dbc"
format_stamp: "Formatted at 2026-09-28 02:34:40 on dist-test-slave-fz47"
server_key: "bdc434e829a450ec3124e31e3c68bdb3"
server_key_iv: "e63821435075504d1ca7a20eeea41b88"
server_key_version: "encryptionkey@0"
I20260928 02:34:40.187255 2797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "c6d2f7123bf44741a51663c40e351dbc"
format_stamp: "Formatted at 2026-09-28 02:34:40 on dist-test-slave-fz47"
server_key: "bdc434e829a450ec3124e31e3c68bdb3"
server_key_iv: "e63821435075504d1ca7a20eeea41b88"
server_key_version: "encryptionkey@0"
I20260928 02:34:40.188376 2797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 02:34:40.189014 2812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:40.189149 2797 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:40.189225 2797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "c6d2f7123bf44741a51663c40e351dbc"
format_stamp: "Formatted at 2026-09-28 02:34:40 on dist-test-slave-fz47"
server_key: "bdc434e829a450ec3124e31e3c68bdb3"
server_key_iv: "e63821435075504d1ca7a20eeea41b88"
server_key_version: "encryptionkey@0"
I20260928 02:34:40.189311 2797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:40.210443 2797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:40.212093 2797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:40.212219 2797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:40.215837 2797 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43141
I20260928 02:34:40.215867 2864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43141 every 8 connection(s)
I20260928 02:34:40.216331 2797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:34:40.216940 2865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:40.219051 2865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6d2f7123bf44741a51663c40e351dbc: Bootstrap starting.
I20260928 02:34:40.219547 2865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6d2f7123bf44741a51663c40e351dbc: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:40.219764 2865 log.cc:824] T 00000000000000000000000000000000 P c6d2f7123bf44741a51663c40e351dbc: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:40.219867 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 2797
I20260928 02:34:40.219942 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:34:40.220103 392 external_mini_cluster.cc:1428] Setting key 97ee1ec2038e7ac61b0ec93416429799
I20260928 02:34:40.220381 2865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6d2f7123bf44741a51663c40e351dbc: No bootstrap required, opened a new log
I20260928 02:34:40.222831 2865 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6d2f7123bf44741a51663c40e351dbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d2f7123bf44741a51663c40e351dbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43141 } }
I20260928 02:34:40.222966 2865 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6d2f7123bf44741a51663c40e351dbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:40.222997 2865 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6d2f7123bf44741a51663c40e351dbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6d2f7123bf44741a51663c40e351dbc, State: Initialized, Role: FOLLOWER
I20260928 02:34:40.223114 2865 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6d2f7123bf44741a51663c40e351dbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d2f7123bf44741a51663c40e351dbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43141 } }
I20260928 02:34:40.223182 2865 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6d2f7123bf44741a51663c40e351dbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:34:40.223212 2865 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6d2f7123bf44741a51663c40e351dbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:34:40.223255 2865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6d2f7123bf44741a51663c40e351dbc [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:40.223855 2865 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6d2f7123bf44741a51663c40e351dbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d2f7123bf44741a51663c40e351dbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43141 } }
I20260928 02:34:40.223964 2865 leader_election.cc:302] T 00000000000000000000000000000000 P c6d2f7123bf44741a51663c40e351dbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6d2f7123bf44741a51663c40e351dbc; no voters:
I20260928 02:34:40.224097 2865 leader_election.cc:288] T 00000000000000000000000000000000 P c6d2f7123bf44741a51663c40e351dbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:34:40.224136 2870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6d2f7123bf44741a51663c40e351dbc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:40.224283 2870 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6d2f7123bf44741a51663c40e351dbc [term 1 LEADER]: Becoming Leader. State: Replica: c6d2f7123bf44741a51663c40e351dbc, State: Running, Role: LEADER
I20260928 02:34:40.224315 2865 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6d2f7123bf44741a51663c40e351dbc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:34:40.224366 2870 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6d2f7123bf44741a51663c40e351dbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d2f7123bf44741a51663c40e351dbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43141 } }
I20260928 02:34:40.224699 2870 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6d2f7123bf44741a51663c40e351dbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6d2f7123bf44741a51663c40e351dbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d2f7123bf44741a51663c40e351dbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43141 } } }
I20260928 02:34:40.224829 2870 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6d2f7123bf44741a51663c40e351dbc [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:40.225000 2870 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6d2f7123bf44741a51663c40e351dbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6d2f7123bf44741a51663c40e351dbc. Latest consensus state: current_term: 1 leader_uuid: "c6d2f7123bf44741a51663c40e351dbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6d2f7123bf44741a51663c40e351dbc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43141 } } }
I20260928 02:34:40.225103 2870 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6d2f7123bf44741a51663c40e351dbc [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:40.225441 2879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:34:40.225865 2879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:34:40.227138 2879 catalog_manager.cc:1347] Generated new cluster ID: 0fdf121989ab490db5475975f670c0b3
I20260928 02:34:40.227195 2879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:34:40 dist-test-slave-fz47 krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:40.232123 2869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:40.220535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40118 (local address 127.0.98.62:43141)
0928 02:34:40.220709 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:40.220717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:40.220741 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:34:40.221512 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:40.221519 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:40.221566 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:40.221673 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:40.222405 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:40.223318 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:40.224304 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:40.224889 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:40.225303 (+ 414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:40.228753 (+ 3450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:40.228757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:40.228758 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:40.228768 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:40.230587 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:40.230958 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:40.230961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:40.230961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:40.231051 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:40.231238 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:40.231239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:40.231239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:40.231492 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:40.231616 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:40.231715 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 02:34:40.231842 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
I20260928 02:34:40.235272 2879 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:34:40.235955 2879 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:34:40.248258 2879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6d2f7123bf44741a51663c40e351dbc: Generated new TSK 0
I20260928 02:34:40.248420 2879 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 02:34:40.273530 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43141
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:40.350040 2891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:40.350224 2891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:40.350244 2891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:40.351310 2891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:40.351344 2891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:40.351368 2891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:34:40.352779 2891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:40.352949 2891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:40.353109 2891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:40.357507 2899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:40.357563 2897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:40.357666 2896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:40.357663 2891 server_base.cc:1034] running on GCE node
I20260928 02:34:40.357956 2891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:40.358160 2891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:40.359292 2891 hybrid_clock.cc:648] HybridClock initialized: now 1790562880359277 us; error 25 us; skew 500 ppm
Sep 28 02:34:40 dist-test-slave-fz47 krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:40.361505 2891 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 02:34:40.361907 2891 webserver.cc:460] Webserver started at http://127.0.98.1:40679/ using document root <none> and password file <none>
I20260928 02:34:40.362102 2891 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:40.362145 2891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:40.362262 2891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:40.363188 2891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/data/instance:
uuid: "97d5b7d172764639a3d7e3444139c294"
format_stamp: "Formatted at 2026-09-28 02:34:40 on dist-test-slave-fz47"
server_key: "d5c8e87de993077abd7b47e7e44529da"
server_key_iv: "809efb477f131d67feb581b1e461c089"
server_key_version: "encryptionkey@0"
I20260928 02:34:40.363500 2891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/wal/instance:
uuid: "97d5b7d172764639a3d7e3444139c294"
format_stamp: "Formatted at 2026-09-28 02:34:40 on dist-test-slave-fz47"
server_key: "d5c8e87de993077abd7b47e7e44529da"
server_key_iv: "809efb477f131d67feb581b1e461c089"
server_key_version: "encryptionkey@0"
I20260928 02:34:40.364624 2891 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:34:40.365217 2906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:40.365357 2891 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:40.365432 2891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "97d5b7d172764639a3d7e3444139c294"
format_stamp: "Formatted at 2026-09-28 02:34:40 on dist-test-slave-fz47"
server_key: "d5c8e87de993077abd7b47e7e44529da"
server_key_iv: "809efb477f131d67feb581b1e461c089"
server_key_version: "encryptionkey@0"
I20260928 02:34:40.365514 2891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:40.387028 2891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:40.388603 2891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:40.388705 2891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:40.388872 2891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:34:40.389140 2891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:40.389171 2891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:40.389199 2891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:40.389223 2891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:40.394708 2891 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46379
I20260928 02:34:40.394753 3019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46379 every 8 connection(s)
I20260928 02:34:40.395035 2891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
I20260928 02:34:40.397336 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 2891
I20260928 02:34:40.397428 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/wal/instance
I20260928 02:34:40.397596 392 external_mini_cluster.cc:1428] Setting key ffe2c257c3b92d5097516dcdce6f03f0
Sep 28 02:34:40 dist-test-slave-fz47 krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:40.404666 2869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:40.395640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44889 (local address 127.0.98.62:43141)
0928 02:34:40.395705 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:40.395707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:40.395888 (+ 181us) server_negotiation.cc:407] Connection header received
0928 02:34:40.396751 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:40.396752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:40.396762 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:40.396809 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:40.397521 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:40.398204 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:40.399560 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:40.399885 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:40.399962 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:40.402074 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:40.402075 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:40.402076 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:40.402084 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:40.403474 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:40.403818 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:40.403819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:40.403820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:40.403879 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:40.404083 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:40.404084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:40.404085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:40.404212 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:40.404286 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:40.404484 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 02:34:40.404565 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:34:40.404698 3022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:40.395628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43141 (local address 127.0.98.1:44889)
0928 02:34:40.395781 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:40.395813 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:40.396672 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:40.396864 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:40.396867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:40.397099 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:40.397375 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:40.397381 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:40.398292 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:40.398293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:40.399457 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:40.399460 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:40.399964 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:40.399964 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:40.400055 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:40.400519 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:40.400527 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:40.401980 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:40.403548 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:40.403553 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:40.403555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:40.403750 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:40.403944 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:40.403945 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:40.403945 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:40.404026 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:40.404268 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:40.404270 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:40.404371 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 02:34:40.404445 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 02:34:40.404551 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
I20260928 02:34:40.405134 3020 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43141
I20260928 02:34:40.405216 3020 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:40.405385 3020 heartbeater.cc:511] Master 127.0.98.62:43141 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 02:34:40.405908 2829 ts_manager.cc:194] Registered new tserver with Master: 97d5b7d172764639a3d7e3444139c294 (127.0.98.1:46379)
I20260928 02:34:40.406522 2829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44889
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 02:34:40.420056 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43141
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:40.495754 3025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:40.495932 3025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:40.495954 3025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:40.497061 3025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:40.497102 3025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:40.497129 3025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:34:40.498581 3025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:40.498782 3025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:40.498970 3025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:34:40.503373 3025 server_base.cc:1034] running on GCE node
W20260928 02:34:40.503409 3030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:40.503492 3033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:40.503352 3031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:40.503762 3025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:40.503974 3025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:40.505107 3025 hybrid_clock.cc:648] HybridClock initialized: now 1790562880505096 us; error 25 us; skew 500 ppm
Sep 28 02:34:40 dist-test-slave-fz47 krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:40.507464 3025 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 02:34:40.507828 3025 webserver.cc:460] Webserver started at http://127.0.98.2:39067/ using document root <none> and password file <none>
I20260928 02:34:40.508028 3025 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:40.508072 3025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:40.508185 3025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:40.509124 3025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/data/instance:
uuid: "881b48e1fb624d0f92d450178740b9ba"
format_stamp: "Formatted at 2026-09-28 02:34:40 on dist-test-slave-fz47"
server_key: "965c41fcd28c59d3f1a43dba0842bbe7"
server_key_iv: "71e6851fc5ce1d871a01562284764aed"
server_key_version: "encryptionkey@0"
I20260928 02:34:40.509456 3025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/wal/instance:
uuid: "881b48e1fb624d0f92d450178740b9ba"
format_stamp: "Formatted at 2026-09-28 02:34:40 on dist-test-slave-fz47"
server_key: "965c41fcd28c59d3f1a43dba0842bbe7"
server_key_iv: "71e6851fc5ce1d871a01562284764aed"
server_key_version: "encryptionkey@0"
I20260928 02:34:40.510567 3025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:34:40.511149 3040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:40.511287 3025 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:34:40.511361 3025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "881b48e1fb624d0f92d450178740b9ba"
format_stamp: "Formatted at 2026-09-28 02:34:40 on dist-test-slave-fz47"
server_key: "965c41fcd28c59d3f1a43dba0842bbe7"
server_key_iv: "71e6851fc5ce1d871a01562284764aed"
server_key_version: "encryptionkey@0"
I20260928 02:34:40.511417 3025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:40.527752 3025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:40.529750 3025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:40.529875 3025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:40.530066 3025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:34:40.530328 3025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:40.530364 3025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:40.530390 3025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:40.530413 3025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:40.535568 3025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33843
I20260928 02:34:40.535627 3153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33843 every 8 connection(s)
I20260928 02:34:40.535884 3025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 02:34:40 dist-test-slave-fz47 krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:40.543738 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 3025
I20260928 02:34:40.543821 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/wal/instance
I20260928 02:34:40.544031 392 external_mini_cluster.cc:1428] Setting key bc766bd6f8a673f9db8e1790226891cd
I20260928 02:34:40.544384 2869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:40.536525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55933 (local address 127.0.98.62:43141)
0928 02:34:40.536578 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:40.536580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:40.536756 (+ 176us) server_negotiation.cc:407] Connection header received
0928 02:34:40.537682 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:40.537683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:40.537694 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:40.537752 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:40.538275 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:40.538768 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:40.539736 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:40.539938 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:40.539989 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:40.541936 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:40.541937 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:40.541938 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:40.541944 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:40.543112 (+ 1168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:40.543458 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:40.543458 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:40.543459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:40.543504 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:40.543709 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:40.543710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:40.543711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:40.543826 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:40.543883 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:40.544207 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 02:34:40.544301 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 02:34:40.544566 3156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:40.536484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43141 (local address 127.0.98.2:55933)
0928 02:34:40.536642 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:40.536680 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:40.537594 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:40.537806 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:40.537810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:40.538038 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:40.538212 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:40.538216 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:40.538846 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:40.538847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:40.539652 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:40.539654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:40.539997 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:40.539997 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:40.540087 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:40.540481 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:40.540488 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:40.541851 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:40.543184 (+ 1333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:40.543189 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:40.543191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:40.543402 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:40.543546 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:40.543547 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:40.543547 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:40.543642 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:40.543905 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:40.543912 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:40.544070 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 02:34:40.544184 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 02:34:40.544335 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260928 02:34:40.544974 3154 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43141
I20260928 02:34:40.545058 3154 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:40.545215 3154 heartbeater.cc:511] Master 127.0.98.62:43141 requested a full tablet report, sending...
I20260928 02:34:40.545570 2829 ts_manager.cc:194] Registered new tserver with Master: 881b48e1fb624d0f92d450178740b9ba (127.0.98.2:33843)
I20260928 02:34:40.545917 2829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55933
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 02:34:40.567821 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43141
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:40.647862 3159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:40.648051 3159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:40.648072 3159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:40.649132 3159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:40.649164 3159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:40.649187 3159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:34:40.650624 3159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:40.650800 3159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:40.650967 3159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:40.655524 3167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:40.655640 3164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:40.655617 3165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:40.655550 3159 server_base.cc:1034] running on GCE node
I20260928 02:34:40.655964 3159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:40.656172 3159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:40.657316 3159 hybrid_clock.cc:648] HybridClock initialized: now 1790562880657297 us; error 35 us; skew 500 ppm
Sep 28 02:34:40 dist-test-slave-fz47 krb5kdc[2784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:40.659636 3159 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 02:34:40.660092 3159 webserver.cc:460] Webserver started at http://127.0.98.3:36285/ using document root <none> and password file <none>
I20260928 02:34:40.660280 3159 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:40.660326 3159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:40.660452 3159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:40.661492 3159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/data/instance:
uuid: "01733d01936d44dd908f9bb5f0eccb79"
format_stamp: "Formatted at 2026-09-28 02:34:40 on dist-test-slave-fz47"
server_key: "11b7709cd9aaf357220f53d6d7cf45d4"
server_key_iv: "26ad1285d590f60a314957e8acb5c4c6"
server_key_version: "encryptionkey@0"
I20260928 02:34:40.661826 3159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/wal/instance:
uuid: "01733d01936d44dd908f9bb5f0eccb79"
format_stamp: "Formatted at 2026-09-28 02:34:40 on dist-test-slave-fz47"
server_key: "11b7709cd9aaf357220f53d6d7cf45d4"
server_key_iv: "26ad1285d590f60a314957e8acb5c4c6"
server_key_version: "encryptionkey@0"
I20260928 02:34:40.663076 3159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:34:40.663728 3174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:40.663895 3159 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:40.663973 3159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "01733d01936d44dd908f9bb5f0eccb79"
format_stamp: "Formatted at 2026-09-28 02:34:40 on dist-test-slave-fz47"
server_key: "11b7709cd9aaf357220f53d6d7cf45d4"
server_key_iv: "26ad1285d590f60a314957e8acb5c4c6"
server_key_version: "encryptionkey@0"
I20260928 02:34:40.664047 3159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:40.680192 3159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:40.681927 3159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:40.682057 3159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:40.682250 3159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:34:40.682529 3159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:40.682561 3159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:40.682588 3159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:40.682605 3159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:40.687919 3159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45367
I20260928 02:34:40.687970 3287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45367 every 8 connection(s)
I20260928 02:34:40.688241 3159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
I20260928 02:34:40.692214 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 3159
I20260928 02:34:40.692327 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/wal/instance
I20260928 02:34:40.692528 392 external_mini_cluster.cc:1428] Setting key 3b9d5ab6f380d97d082579fcfde56ffe
Sep 28 02:34:40 dist-test-slave-fz47 krb5kdc[2784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:40.696833 2869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:40.688865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35573 (local address 127.0.98.62:43141)
0928 02:34:40.688907 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:40.688909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:40.689090 (+ 181us) server_negotiation.cc:407] Connection header received
0928 02:34:40.690021 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:40.690022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:40.690033 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:40.690088 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:40.690642 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:40.691150 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:40.692262 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:40.692543 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:40.692609 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:40.694720 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:40.694721 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:40.694721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:40.694727 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:40.695804 (+ 1077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:40.696152 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:40.696152 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:40.696153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:40.696201 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:40.696375 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:40.696376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:40.696376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:40.696464 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:40.696508 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:40.696709 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 02:34:40.696763 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:34:40.697005 3290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:40.688826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43141 (local address 127.0.98.3:35573)
0928 02:34:40.688972 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:40.689009 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:40.689929 (+ 920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:40.690159 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:40.690162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:40.690400 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:40.690581 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:40.690585 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:40.691226 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:40.691227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:40.692187 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:40.692190 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:40.692633 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:40.692634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:40.692813 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:40.693323 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:40.693333 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:40.694632 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:40.695887 (+ 1255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:40.695893 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:40.695894 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:40.696094 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:40.696250 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:40.696250 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:40.696250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:40.696333 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:40.696504 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:40.696507 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:40.696610 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 02:34:40.696693 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 02:34:40.696801 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
I20260928 02:34:40.697424 3288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43141
I20260928 02:34:40.697508 3288 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:40.697639 3288 heartbeater.cc:511] Master 127.0.98.62:43141 requested a full tablet report, sending...
I20260928 02:34:40.697956 2829 ts_manager.cc:194] Registered new tserver with Master: 01733d01936d44dd908f9bb5f0eccb79 (127.0.98.3:45367)
I20260928 02:34:40.698277 2829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35573
I20260928 02:34:40.703974 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 02:34:40.713685 2869 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:34:40.712837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40134 (local address 127.0.98.62:43141)
0928 02:34:40.712901 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:40.712903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:40.712986 (+ 83us) server_negotiation.cc:407] Connection header received
0928 02:34:40.713050 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:40.713050 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:40.713059 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:40.713104 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:40.713630 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40134: BlockingRecv error: recv got EOF from 127.0.0.1:40134 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 02:34:40.714541 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 2891
I20260928 02:34:40.718200 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 3025
I20260928 02:34:40.721808 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 3159
I20260928 02:34:40.725479 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 2797
2026-09-28T02:34:40Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1679 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:34:40 dist-test-slave-fz47 krb5kdc[3300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:34:40 dist-test-slave-fz47 krb5kdc[3300](info): set up 2 sockets
Sep 28 02:34:40 dist-test-slave-fz47 krb5kdc[3300](info): commencing operation
krb5kdc: starting...
W20260928 02:34:41.664793 392 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:34:41 dist-test-slave-fz47 krb5kdc[3300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:34:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:34:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 02:34:41.775429 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33893
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:41.852805 3313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:41.853003 3313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:41.853029 3313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:41.854188 3313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:34:41.854231 3313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:41.854249 3313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:41.854264 3313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:34:41.854279 3313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:34:41.855762 3313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:41.856001 3313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:41.856189 3313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:41.860544 3319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:41.860549 3321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:41.860569 3318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:41.860762 3313 server_base.cc:1034] running on GCE node
I20260928 02:34:41.861004 3313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:41.861246 3313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:41.862408 3313 hybrid_clock.cc:648] HybridClock initialized: now 1790562881862397 us; error 29 us; skew 500 ppm
Sep 28 02:34:41 dist-test-slave-fz47 krb5kdc[3300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:41.867372 3313 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:34:41.867761 3313 webserver.cc:460] Webserver started at http://127.0.98.62:46173/ using document root <none> and password file <none>
I20260928 02:34:41.867959 3313 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:41.868000 3313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:41.868119 3313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:41.869118 3313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "89f559f350fd41cb8202697329a33361"
format_stamp: "Formatted at 2026-09-28 02:34:41 on dist-test-slave-fz47"
server_key: "2adb28607e0128a660cc32888719117e"
server_key_iv: "7026f5b1713cb89f33db80abface0646"
server_key_version: "encryptionkey@0"
I20260928 02:34:41.869450 3313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "89f559f350fd41cb8202697329a33361"
format_stamp: "Formatted at 2026-09-28 02:34:41 on dist-test-slave-fz47"
server_key: "2adb28607e0128a660cc32888719117e"
server_key_iv: "7026f5b1713cb89f33db80abface0646"
server_key_version: "encryptionkey@0"
I20260928 02:34:41.870565 3313 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:34:41.871168 3328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:41.871313 3313 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:34:41.871384 3313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "89f559f350fd41cb8202697329a33361"
format_stamp: "Formatted at 2026-09-28 02:34:41 on dist-test-slave-fz47"
server_key: "2adb28607e0128a660cc32888719117e"
server_key_iv: "7026f5b1713cb89f33db80abface0646"
server_key_version: "encryptionkey@0"
I20260928 02:34:41.871447 3313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:41.879628 3313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:41.881529 3313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:41.881645 3313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:41.885118 3313 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33893
I20260928 02:34:41.885146 3380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33893 every 8 connection(s)
I20260928 02:34:41.885484 3313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:34:41.886241 3381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:41.888281 3381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89f559f350fd41cb8202697329a33361: Bootstrap starting.
I20260928 02:34:41.888769 3381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89f559f350fd41cb8202697329a33361: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:41.888995 3381 log.cc:824] T 00000000000000000000000000000000 P 89f559f350fd41cb8202697329a33361: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:41.889279 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 3313
I20260928 02:34:41.889382 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:34:41.889533 392 external_mini_cluster.cc:1428] Setting key 00f1024a542b028c4ae618a2ad333b54
I20260928 02:34:41.889708 3381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89f559f350fd41cb8202697329a33361: No bootstrap required, opened a new log
I20260928 02:34:41.891903 3381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89f559f350fd41cb8202697329a33361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f559f350fd41cb8202697329a33361" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33893 } }
I20260928 02:34:41.892032 3381 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89f559f350fd41cb8202697329a33361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:41.892062 3381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89f559f350fd41cb8202697329a33361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f559f350fd41cb8202697329a33361, State: Initialized, Role: FOLLOWER
I20260928 02:34:41.892179 3381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89f559f350fd41cb8202697329a33361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f559f350fd41cb8202697329a33361" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33893 } }
I20260928 02:34:41.892248 3381 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89f559f350fd41cb8202697329a33361 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:34:41.892277 3381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89f559f350fd41cb8202697329a33361 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:34:41.892320 3381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89f559f350fd41cb8202697329a33361 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:41.892813 3381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89f559f350fd41cb8202697329a33361 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f559f350fd41cb8202697329a33361" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33893 } }
I20260928 02:34:41.892904 3381 leader_election.cc:302] T 00000000000000000000000000000000 P 89f559f350fd41cb8202697329a33361 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89f559f350fd41cb8202697329a33361; no voters:
I20260928 02:34:41.893041 3381 leader_election.cc:288] T 00000000000000000000000000000000 P 89f559f350fd41cb8202697329a33361 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:34:41.893083 3386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89f559f350fd41cb8202697329a33361 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:41.893214 3386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89f559f350fd41cb8202697329a33361 [term 1 LEADER]: Becoming Leader. State: Replica: 89f559f350fd41cb8202697329a33361, State: Running, Role: LEADER
I20260928 02:34:41.893296 3381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89f559f350fd41cb8202697329a33361 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:34:41.893368 3386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89f559f350fd41cb8202697329a33361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f559f350fd41cb8202697329a33361" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33893 } }
I20260928 02:34:41.893723 3386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89f559f350fd41cb8202697329a33361 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89f559f350fd41cb8202697329a33361. Latest consensus state: current_term: 1 leader_uuid: "89f559f350fd41cb8202697329a33361" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f559f350fd41cb8202697329a33361" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33893 } } }
I20260928 02:34:41.893872 3386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89f559f350fd41cb8202697329a33361 [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:41.894109 3394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:34:41.894611 3394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:34:41.894762 3388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89f559f350fd41cb8202697329a33361 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89f559f350fd41cb8202697329a33361" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f559f350fd41cb8202697329a33361" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33893 } } }
I20260928 02:34:41.895624 3388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89f559f350fd41cb8202697329a33361 [sys.catalog]: This master's current role is: LEADER
Sep 28 02:34:41 dist-test-slave-fz47 krb5kdc[3300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:41.896802 3394 catalog_manager.cc:1347] Generated new cluster ID: 5b76dffe84964e439bdbeb137b550f9f
I20260928 02:34:41.896874 3394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:34:41.899817 3385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:41.889953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43390 (local address 127.0.98.62:33893)
0928 02:34:41.890125 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:41.890133 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:41.890155 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:34:41.890878 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:41.890885 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:41.890931 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:41.891051 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:41.891627 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:41.892605 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:41.893543 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:41.894095 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:41.894198 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:41.896757 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:41.896760 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:41.896761 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:41.896794 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:41.898401 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:41.898712 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:41.898715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:41.898716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:41.898824 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:41.898996 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:41.898996 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:41.898997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:41.899227 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:41.899365 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:41.899476 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 02:34:41.899605 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
I20260928 02:34:41.900636 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:34:41.906837 3394 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:34:41.907524 3394 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:34:41.919050 3394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89f559f350fd41cb8202697329a33361: Generated new TSK 0
I20260928 02:34:41.919235 3394 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:34:41.919459 3385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:41.915810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43400 (local address 127.0.98.62:33893)
0928 02:34:41.915852 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:41.915854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:41.915943 (+ 89us) server_negotiation.cc:407] Connection header received
0928 02:34:41.916006 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:41.916007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:41.916017 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:41.916064 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:41.916434 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:41.917385 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:41.918104 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:41.918330 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:41.918384 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:41.919012 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:41.919013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:41.919014 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:34:41.919213 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:41.919233 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:34:41.919240 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:41.919288 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:41.919362 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 02:34:41.919388 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:34:41.921882 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 3313
2026-09-28T02:34:41Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1194 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:34:41 dist-test-slave-fz47 krb5kdc[3414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:34:41 dist-test-slave-fz47 krb5kdc[3414](info): set up 2 sockets
Sep 28 02:34:41 dist-test-slave-fz47 krb5kdc[3414](info): commencing operation
krb5kdc: starting...
W20260928 02:34:42.860833 392 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:34:42 dist-test-slave-fz47 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:34:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:34:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 02:34:42.963738 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37707
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:43.040390 3427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:43.040585 3427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:43.040614 3427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:43.041713 3427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:34:43.041754 3427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:43.041767 3427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:43.041778 3427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:34:43.041788 3427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:34:43.043164 3427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:43.043340 3427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:43.043506 3427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:43.047958 3432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:43.047981 3433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:43.048046 3427 server_base.cc:1034] running on GCE node
W20260928 02:34:43.047967 3435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:43.048367 3427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:43.048622 3427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:43.049767 3427 hybrid_clock.cc:648] HybridClock initialized: now 1790562883049752 us; error 28 us; skew 500 ppm
Sep 28 02:34:43 dist-test-slave-fz47 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:43.052078 3427 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:34:43.052481 3427 webserver.cc:460] Webserver started at http://127.0.98.62:40977/ using document root <none> and password file <none>
I20260928 02:34:43.052683 3427 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:43.052731 3427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:43.052850 3427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:43.053881 3427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "5f538f6a4fa64149b8fa50561f118e25"
format_stamp: "Formatted at 2026-09-28 02:34:43 on dist-test-slave-fz47"
server_key: "7b61d4189bcfbf940d778c3948660900"
server_key_iv: "cb4ffc33b5cf3a7a76a5cb643d69f340"
server_key_version: "encryptionkey@0"
I20260928 02:34:43.054183 3427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "5f538f6a4fa64149b8fa50561f118e25"
format_stamp: "Formatted at 2026-09-28 02:34:43 on dist-test-slave-fz47"
server_key: "7b61d4189bcfbf940d778c3948660900"
server_key_iv: "cb4ffc33b5cf3a7a76a5cb643d69f340"
server_key_version: "encryptionkey@0"
I20260928 02:34:43.055322 3427 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:34:43.055902 3442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:43.056043 3427 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:43.056120 3427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "5f538f6a4fa64149b8fa50561f118e25"
format_stamp: "Formatted at 2026-09-28 02:34:43 on dist-test-slave-fz47"
server_key: "7b61d4189bcfbf940d778c3948660900"
server_key_iv: "cb4ffc33b5cf3a7a76a5cb643d69f340"
server_key_version: "encryptionkey@0"
I20260928 02:34:43.056175 3427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:43.072764 3427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:43.074486 3427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:43.074622 3427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:43.078173 3427 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37707
I20260928 02:34:43.078207 3494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37707 every 8 connection(s)
I20260928 02:34:43.078518 3427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:34:43.079417 3495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:43.081575 3495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f538f6a4fa64149b8fa50561f118e25: Bootstrap starting.
I20260928 02:34:43.082085 3495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f538f6a4fa64149b8fa50561f118e25: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:43.082329 3495 log.cc:824] T 00000000000000000000000000000000 P 5f538f6a4fa64149b8fa50561f118e25: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:43.082917 3495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f538f6a4fa64149b8fa50561f118e25: No bootstrap required, opened a new log
I20260928 02:34:43.084403 3495 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f538f6a4fa64149b8fa50561f118e25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f538f6a4fa64149b8fa50561f118e25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37707 } }
I20260928 02:34:43.084535 3495 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f538f6a4fa64149b8fa50561f118e25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:43.084556 3495 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f538f6a4fa64149b8fa50561f118e25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f538f6a4fa64149b8fa50561f118e25, State: Initialized, Role: FOLLOWER
I20260928 02:34:43.084697 3495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f538f6a4fa64149b8fa50561f118e25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f538f6a4fa64149b8fa50561f118e25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37707 } }
I20260928 02:34:43.084769 3495 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f538f6a4fa64149b8fa50561f118e25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:34:43.084789 3495 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f538f6a4fa64149b8fa50561f118e25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:34:43.084816 3495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f538f6a4fa64149b8fa50561f118e25 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:43.085580 3495 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f538f6a4fa64149b8fa50561f118e25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f538f6a4fa64149b8fa50561f118e25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37707 } }
I20260928 02:34:43.085661 3495 leader_election.cc:302] T 00000000000000000000000000000000 P 5f538f6a4fa64149b8fa50561f118e25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f538f6a4fa64149b8fa50561f118e25; no voters:
I20260928 02:34:43.085803 3495 leader_election.cc:288] T 00000000000000000000000000000000 P 5f538f6a4fa64149b8fa50561f118e25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:34:43.085845 3498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f538f6a4fa64149b8fa50561f118e25 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:43.085983 3498 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f538f6a4fa64149b8fa50561f118e25 [term 1 LEADER]: Becoming Leader. State: Replica: 5f538f6a4fa64149b8fa50561f118e25, State: Running, Role: LEADER
I20260928 02:34:43.086036 3495 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f538f6a4fa64149b8fa50561f118e25 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:34:43.086071 3498 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f538f6a4fa64149b8fa50561f118e25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f538f6a4fa64149b8fa50561f118e25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37707 } }
I20260928 02:34:43.086434 3499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f538f6a4fa64149b8fa50561f118e25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f538f6a4fa64149b8fa50561f118e25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f538f6a4fa64149b8fa50561f118e25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37707 } } }
I20260928 02:34:43.086457 3500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f538f6a4fa64149b8fa50561f118e25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f538f6a4fa64149b8fa50561f118e25. Latest consensus state: current_term: 1 leader_uuid: "5f538f6a4fa64149b8fa50561f118e25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f538f6a4fa64149b8fa50561f118e25" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37707 } } }
I20260928 02:34:43.086560 3499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f538f6a4fa64149b8fa50561f118e25 [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:43.086824 3500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f538f6a4fa64149b8fa50561f118e25 [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:43.086922 3507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:34:43.087388 3507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:34:43.087707 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 3427
I20260928 02:34:43.087793 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:34:43.087960 392 external_mini_cluster.cc:1428] Setting key 514bfe32b1e595be275da613624c232a
I20260928 02:34:43.088972 3507 catalog_manager.cc:1347] Generated new cluster ID: 01055ec0778d4e2bae03f6acf8395cc9
I20260928 02:34:43.089015 3507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:34:43 dist-test-slave-fz47 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:43.099257 3517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:43.088378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53448 (local address 127.0.98.62:37707)
0928 02:34:43.088525 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:43.088533 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:43.088621 (+ 88us) server_negotiation.cc:407] Connection header received
0928 02:34:43.088753 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:43.088760 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:43.088817 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:43.089008 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:43.089717 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:43.091125 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:43.092120 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:43.092726 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:43.092832 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:43.095568 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:43.095572 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:43.095572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:43.095583 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:43.097441 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:43.097836 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:43.097838 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:43.097838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:43.097936 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:43.098115 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:43.098116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:43.098116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:43.098380 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:43.098519 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:43.098611 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 02:34:43.099025 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20260928 02:34:43.103359 3507 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:34:43.103888 3507 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:34:43.109017 3507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f538f6a4fa64149b8fa50561f118e25: Generated new TSK 0
I20260928 02:34:43.109179 3507 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 02:34:43.135848 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37707
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:43.213512 3521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:43.213712 3521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:43.213742 3521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:43.214879 3521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:43.214917 3521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:43.214951 3521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:34:43.216337 3521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:43.216557 3521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:43.216758 3521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:43.221326 3527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:43.221318 3526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:43.221340 3529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:43.221561 3521 server_base.cc:1034] running on GCE node
I20260928 02:34:43.221750 3521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:43.221940 3521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:43.223094 3521 hybrid_clock.cc:648] HybridClock initialized: now 1790562883223075 us; error 29 us; skew 500 ppm
Sep 28 02:34:43 dist-test-slave-fz47 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:43.225370 3521 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 02:34:43.225764 3521 webserver.cc:460] Webserver started at http://127.0.98.1:41617/ using document root <none> and password file <none>
I20260928 02:34:43.225946 3521 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:43.225983 3521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:43.226068 3521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:43.227005 3521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/data/instance:
uuid: "e9e40e201cef471297c939d60b93f6b1"
format_stamp: "Formatted at 2026-09-28 02:34:43 on dist-test-slave-fz47"
server_key: "6f2e9356873eaa5b0953439066727f1b"
server_key_iv: "fb4b8051c6683847db3b5a5e936575c0"
server_key_version: "encryptionkey@0"
I20260928 02:34:43.227280 3521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e9e40e201cef471297c939d60b93f6b1"
format_stamp: "Formatted at 2026-09-28 02:34:43 on dist-test-slave-fz47"
server_key: "6f2e9356873eaa5b0953439066727f1b"
server_key_iv: "fb4b8051c6683847db3b5a5e936575c0"
server_key_version: "encryptionkey@0"
I20260928 02:34:43.228365 3521 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:34:43.228956 3536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:43.229102 3521 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:43.229164 3521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "e9e40e201cef471297c939d60b93f6b1"
format_stamp: "Formatted at 2026-09-28 02:34:43 on dist-test-slave-fz47"
server_key: "6f2e9356873eaa5b0953439066727f1b"
server_key_iv: "fb4b8051c6683847db3b5a5e936575c0"
server_key_version: "encryptionkey@0"
I20260928 02:34:43.229228 3521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:43.259662 3521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:43.261428 3521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:43.261545 3521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:43.261803 3521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:34:43.262267 3521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:43.262317 3521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:43.262389 3521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:43.262422 3521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:43.268795 3521 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46821
I20260928 02:34:43.268868 3649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46821 every 8 connection(s)
I20260928 02:34:43.269129 3521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
I20260928 02:34:43.270413 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 3521
I20260928 02:34:43.270503 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-0/wal/instance
I20260928 02:34:43.270725 392 external_mini_cluster.cc:1428] Setting key 4504b97cad148071237969ba4c585531
Sep 28 02:34:43 dist-test-slave-fz47 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 02:34:43.278354 3517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:43.269817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55053 (local address 127.0.98.62:37707)
0928 02:34:43.269892 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:43.269894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:43.270050 (+ 156us) server_negotiation.cc:407] Connection header received
0928 02:34:43.270537 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:43.270539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:43.270550 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:43.270601 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:43.271675 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:43.272158 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:43.273114 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:43.273390 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:43.273444 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:43.275662 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:43.275663 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:43.275664 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:43.275670 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:43.276967 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:43.277482 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:43.277482 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:43.277483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:43.277534 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:43.277787 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:43.277789 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:43.277789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:43.277894 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:43.277942 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:43.278223 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 02:34:43.278290 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:34:43.278498 3652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:43.269799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37707 (local address 127.0.98.1:55053)
0928 02:34:43.269941 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:43.269977 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:43.270441 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:43.270673 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:43.270678 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:43.271373 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:43.271591 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:43.271596 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:43.272235 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:43.272236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:43.273038 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:43.273040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:43.273457 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:43.273458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:43.273582 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:43.274027 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:43.274035 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:43.275576 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:43.277058 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:43.277065 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:43.277068 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:43.277380 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:43.277594 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:43.277594 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:43.277595 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:43.277721 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:43.277947 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:43.277954 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:43.278118 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 02:34:43.278204 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 02:34:43.278319 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260928 02:34:43.278915 3650 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37707
I20260928 02:34:43.278997 3650 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:43.279141 3650 heartbeater.cc:511] Master 127.0.98.62:37707 requested a full tablet report, sending...
I20260928 02:34:43.279640 3459 ts_manager.cc:194] Registered new tserver with Master: e9e40e201cef471297c939d60b93f6b1 (127.0.98.1:46821)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260928 02:34:43.280331 3459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55053
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 02:34:43.292037 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37707
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:43.369200 3655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:43.369395 3655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:43.369414 3655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:43.370481 3655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:43.370512 3655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:43.370535 3655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:34:43.371894 3655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:43.372064 3655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:43.372237 3655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:43.376809 3660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:43.376880 3655 server_base.cc:1034] running on GCE node
W20260928 02:34:43.377002 3663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:43.377056 3661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:43.377324 3655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:43.377566 3655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:43.378714 3655 hybrid_clock.cc:648] HybridClock initialized: now 1790562883378698 us; error 32 us; skew 500 ppm
Sep 28 02:34:43 dist-test-slave-fz47 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:43.381601 3655 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 02:34:43.382009 3655 webserver.cc:460] Webserver started at http://127.0.98.2:36297/ using document root <none> and password file <none>
I20260928 02:34:43.382217 3655 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:43.382264 3655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:43.382380 3655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:43.383343 3655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/data/instance:
uuid: "0dad5347346b4c00b54090c4463388f0"
format_stamp: "Formatted at 2026-09-28 02:34:43 on dist-test-slave-fz47"
server_key: "8f833cac8c055cb95a5a192f52c1942a"
server_key_iv: "546ca1f8a7d651422e7da94f8171396f"
server_key_version: "encryptionkey@0"
I20260928 02:34:43.383648 3655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0dad5347346b4c00b54090c4463388f0"
format_stamp: "Formatted at 2026-09-28 02:34:43 on dist-test-slave-fz47"
server_key: "8f833cac8c055cb95a5a192f52c1942a"
server_key_iv: "546ca1f8a7d651422e7da94f8171396f"
server_key_version: "encryptionkey@0"
I20260928 02:34:43.384807 3655 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:34:43.385475 3670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:43.385630 3655 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:43.385715 3655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "0dad5347346b4c00b54090c4463388f0"
format_stamp: "Formatted at 2026-09-28 02:34:43 on dist-test-slave-fz47"
server_key: "8f833cac8c055cb95a5a192f52c1942a"
server_key_iv: "546ca1f8a7d651422e7da94f8171396f"
server_key_version: "encryptionkey@0"
I20260928 02:34:43.385778 3655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:43.417030 3655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:43.419137 3655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:43.419277 3655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:43.419486 3655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:34:43.419765 3655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:43.419797 3655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:43.419826 3655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:43.419849 3655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:43.425087 3655 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37271
I20260928 02:34:43.425153 3783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37271 every 8 connection(s)
I20260928 02:34:43.425451 3655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
I20260928 02:34:43.426177 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 3655
I20260928 02:34:43.426280 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-1/wal/instance
I20260928 02:34:43.426471 392 external_mini_cluster.cc:1428] Setting key a5a91686a62f76937070330578ebbe00
Sep 28 02:34:43 dist-test-slave-fz47 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 02:34:43.434371 3517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:43.426125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33049 (local address 127.0.98.62:37707)
0928 02:34:43.426185 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:43.426187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:43.426359 (+ 172us) server_negotiation.cc:407] Connection header received
0928 02:34:43.426914 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:43.426915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:43.426926 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:43.426989 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:43.427886 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:43.428508 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:43.429706 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:43.429911 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:43.429966 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:43.431902 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:43.431903 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:43.431904 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:43.431911 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:43.433140 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:43.433525 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:43.433526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:43.433527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:43.433576 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:43.433800 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:43.433801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:43.433801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:43.433903 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:43.433956 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:43.434197 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 02:34:43.434281 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 02:34:43.434671 3786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:43.426095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37707 (local address 127.0.98.2:33049)
0928 02:34:43.426244 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:43.426283 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:43.426819 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:43.427057 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:43.427062 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:43.427604 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:43.427790 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:43.427795 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:43.428584 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:43.428585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:43.429630 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:43.429633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:43.429977 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:43.429978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:43.430103 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:43.430526 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:43.430534 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:43.431825 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:43.433209 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:43.433216 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:43.433218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:43.433462 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:43.433627 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:43.433628 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:43.433628 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:43.433746 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:43.433954 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:43.433961 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:43.434087 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 02:34:43.434170 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 02:34:43.434338 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20260928 02:34:43.435155 3784 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37707
I20260928 02:34:43.435261 3784 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:43.435425 3784 heartbeater.cc:511] Master 127.0.98.62:37707 requested a full tablet report, sending...
I20260928 02:34:43.435745 3459 ts_manager.cc:194] Registered new tserver with Master: 0dad5347346b4c00b54090c4463388f0 (127.0.98.2:37271)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260928 02:34:43.436134 3459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33049
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 02:34:43.447525 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37707
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:43.523128 3789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:43.523306 3789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:43.523326 3789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:43.524415 3789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:43.524451 3789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:43.524479 3789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:34:43.525847 3789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:43.526041 3789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:43.526233 3789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:43.530579 3795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:43.530575 3797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:43.530575 3794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:43.530790 3789 server_base.cc:1034] running on GCE node
I20260928 02:34:43.530961 3789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:43.531178 3789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:43.532318 3789 hybrid_clock.cc:648] HybridClock initialized: now 1790562883532307 us; error 34 us; skew 500 ppm
Sep 28 02:34:43 dist-test-slave-fz47 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:43.534567 3789 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 02:34:43.534958 3789 webserver.cc:460] Webserver started at http://127.0.98.3:41397/ using document root <none> and password file <none>
I20260928 02:34:43.535135 3789 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:43.535176 3789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:43.535290 3789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:43.536224 3789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/data/instance:
uuid: "9fca1ee31fe04515ad9d307a69310256"
format_stamp: "Formatted at 2026-09-28 02:34:43 on dist-test-slave-fz47"
server_key: "199bcfaaf4ff570c5e8464120bf8128f"
server_key_iv: "6cca011ca89243d0e9e4a23c7a7e5c40"
server_key_version: "encryptionkey@0"
I20260928 02:34:43.536530 3789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9fca1ee31fe04515ad9d307a69310256"
format_stamp: "Formatted at 2026-09-28 02:34:43 on dist-test-slave-fz47"
server_key: "199bcfaaf4ff570c5e8464120bf8128f"
server_key_iv: "6cca011ca89243d0e9e4a23c7a7e5c40"
server_key_version: "encryptionkey@0"
I20260928 02:34:43.537678 3789 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:34:43.538267 3804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:43.538406 3789 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:43.538477 3789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "9fca1ee31fe04515ad9d307a69310256"
format_stamp: "Formatted at 2026-09-28 02:34:43 on dist-test-slave-fz47"
server_key: "199bcfaaf4ff570c5e8464120bf8128f"
server_key_iv: "6cca011ca89243d0e9e4a23c7a7e5c40"
server_key_version: "encryptionkey@0"
I20260928 02:34:43.538532 3789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:43.557422 3789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:43.559310 3789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:43.559432 3789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:43.559608 3789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:34:43.559890 3789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:43.559921 3789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:43.559947 3789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:43.559970 3789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:43.565271 3789 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39987
I20260928 02:34:43.565342 3917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39987 every 8 connection(s)
I20260928 02:34:43.565598 3789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 02:34:43 dist-test-slave-fz47 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:43.571408 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 3789
I20260928 02:34:43.571509 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790562862804661-392-0/minicluster-data/ts-2/wal/instance
I20260928 02:34:43.571705 392 external_mini_cluster.cc:1428] Setting key 33b1e580ded57d2674ae4e3821d238a5
I20260928 02:34:43.573664 3517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:43.566225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46067 (local address 127.0.98.62:37707)
0928 02:34:43.566299 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:43.566302 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:43.566455 (+ 153us) server_negotiation.cc:407] Connection header received
0928 02:34:43.566895 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:43.566895 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:43.566905 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:43.566953 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:43.567773 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:43.568254 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:43.569157 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:43.569356 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:43.569411 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:43.571237 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:43.571238 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:43.571239 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:43.571247 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:43.572638 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:43.572963 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:43.572963 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:43.572964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:43.573008 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:43.573157 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:43.573157 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:43.573157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:43.573246 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:43.573332 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:43.573537 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 02:34:43.573603 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:34:43.573781 3920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:43.566198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37707 (local address 127.0.98.3:46067)
0928 02:34:43.566349 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:43.566381 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:43.566822 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:43.567005 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:43.567009 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:43.567511 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:43.567689 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:43.567693 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:43.568315 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:43.568316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:43.569086 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:43.569088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:43.569414 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:43.569415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:43.569504 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:43.569932 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:43.569941 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:43.571157 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:43.572695 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:43.572700 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:43.572702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:43.572910 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:43.573042 (+ 132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:43.573043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:43.573043 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:43.573119 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:43.573322 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:43.573325 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:43.573423 (+ 98us) client_negotiation.cc:769] Sending connection context
0928 02:34:43.573498 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 02:34:43.573605 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
I20260928 02:34:43.574138 3918 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37707
I20260928 02:34:43.574213 3918 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:43.574365 3918 heartbeater.cc:511] Master 127.0.98.62:37707 requested a full tablet report, sending...
I20260928 02:34:43.574657 3459 ts_manager.cc:194] Registered new tserver with Master: 9fca1ee31fe04515ad9d307a69310256 (127.0.98.3:39987)
I20260928 02:34:43.574995 3459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46067
I20260928 02:34:43.583142 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:34:43 dist-test-slave-fz47 krb5kdc[3414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562883, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 02:34:43 dist-test-slave-fz47 krb5kdc[3414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562883, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:43.603304 3517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:43.597185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53464 (local address 127.0.98.62:37707)
0928 02:34:43.597242 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:43.597244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:43.597348 (+ 104us) server_negotiation.cc:407] Connection header received
0928 02:34:43.597406 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:43.597407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:43.597417 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:43.597456 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:43.597977 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:43.598455 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:43.599252 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:43.599430 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:43.599481 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:43.601472 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:43.601473 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:43.601474 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:43.601480 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:43.602486 (+ 1006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:43.602764 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:43.602764 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:43.602764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:43.602807 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:43.602933 (+ 126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:43.602933 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:43.602934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:43.603051 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:43.603116 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:43.603186 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 02:34:43.603243 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260928 02:34:43.603499 3459 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53464
I20260928 02:34:43.603825 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 3521
I20260928 02:34:43.607864 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 3655
I20260928 02:34:43.611454 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 3789
I20260928 02:34:43.614884 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 3427
2026-09-28T02:34:43Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1694 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:34:43 dist-test-slave-fz47 krb5kdc[3930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:34:43 dist-test-slave-fz47 krb5kdc[3930](info): set up 2 sockets
Sep 28 02:34:43 dist-test-slave-fz47 krb5kdc[3930](info): commencing operation
krb5kdc: starting...
W20260928 02:34:44.556883 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:34:44 dist-test-slave-fz47 krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:34:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:34:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 02:34:44.662168 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45427
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:44.741475 3943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:44.741672 3943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:44.741691 3943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:44.742812 3943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:34:44.742851 3943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:44.742862 3943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:44.742873 3943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:34:44.742892 3943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:34:44.744298 3943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:44.744524 3943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:44.744712 3943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:44.749172 3948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:44.749250 3943 server_base.cc:1034] running on GCE node
W20260928 02:34:44.749347 3951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:44.749387 3949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:44.749665 3943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:44.749897 3943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:44.751044 3943 hybrid_clock.cc:648] HybridClock initialized: now 1790562884751028 us; error 32 us; skew 500 ppm
Sep 28 02:34:44 dist-test-slave-fz47 krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:44.753211 3943 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:34:44.753656 3943 webserver.cc:460] Webserver started at http://127.0.98.62:33817/ using document root <none> and password file <none>
I20260928 02:34:44.753845 3943 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:44.753890 3943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:44.754005 3943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:44.754977 3943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "68c4966fcf3e41b9a0a29a436fda8fb1"
format_stamp: "Formatted at 2026-09-28 02:34:44 on dist-test-slave-fz47"
server_key: "187979b5672cf43d160653afcbd475a9"
server_key_iv: "404ff704461642b8aa3031cdbc216617"
server_key_version: "encryptionkey@0"
I20260928 02:34:44.755281 3943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "68c4966fcf3e41b9a0a29a436fda8fb1"
format_stamp: "Formatted at 2026-09-28 02:34:44 on dist-test-slave-fz47"
server_key: "187979b5672cf43d160653afcbd475a9"
server_key_iv: "404ff704461642b8aa3031cdbc216617"
server_key_version: "encryptionkey@0"
I20260928 02:34:44.756419 3943 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 02:34:44.757018 3958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:44.757165 3943 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:44.757241 3943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "68c4966fcf3e41b9a0a29a436fda8fb1"
format_stamp: "Formatted at 2026-09-28 02:34:44 on dist-test-slave-fz47"
server_key: "187979b5672cf43d160653afcbd475a9"
server_key_iv: "404ff704461642b8aa3031cdbc216617"
server_key_version: "encryptionkey@0"
I20260928 02:34:44.757328 3943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:44.763945 3943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:44.765491 3943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:44.765607 3943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:44.769068 3943 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45427
I20260928 02:34:44.769107 4010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45427 every 8 connection(s)
I20260928 02:34:44.769440 3943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:34:44.770213 4011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:44.772262 4011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68c4966fcf3e41b9a0a29a436fda8fb1: Bootstrap starting.
I20260928 02:34:44.772765 4011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68c4966fcf3e41b9a0a29a436fda8fb1: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:44.773010 4011 log.cc:824] T 00000000000000000000000000000000 P 68c4966fcf3e41b9a0a29a436fda8fb1: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:44.773635 4011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68c4966fcf3e41b9a0a29a436fda8fb1: No bootstrap required, opened a new log
I20260928 02:34:44.775127 4011 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68c4966fcf3e41b9a0a29a436fda8fb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c4966fcf3e41b9a0a29a436fda8fb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45427 } }
I20260928 02:34:44.775250 4011 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68c4966fcf3e41b9a0a29a436fda8fb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:44.775270 4011 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68c4966fcf3e41b9a0a29a436fda8fb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68c4966fcf3e41b9a0a29a436fda8fb1, State: Initialized, Role: FOLLOWER
I20260928 02:34:44.775401 4011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68c4966fcf3e41b9a0a29a436fda8fb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c4966fcf3e41b9a0a29a436fda8fb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45427 } }
I20260928 02:34:44.775461 4011 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68c4966fcf3e41b9a0a29a436fda8fb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:34:44.775481 4011 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68c4966fcf3e41b9a0a29a436fda8fb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:34:44.775524 4011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68c4966fcf3e41b9a0a29a436fda8fb1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:44.775992 4011 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68c4966fcf3e41b9a0a29a436fda8fb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c4966fcf3e41b9a0a29a436fda8fb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45427 } }
I20260928 02:34:44.776007 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 3943
I20260928 02:34:44.776084 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:34:44.776070 4011 leader_election.cc:302] T 00000000000000000000000000000000 P 68c4966fcf3e41b9a0a29a436fda8fb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68c4966fcf3e41b9a0a29a436fda8fb1; no voters:
I20260928 02:34:44.776219 4011 leader_election.cc:288] T 00000000000000000000000000000000 P 68c4966fcf3e41b9a0a29a436fda8fb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:34:44.776261 392 external_mini_cluster.cc:1428] Setting key 3253539f4d06de173c2c7985e1fe5f83
I20260928 02:34:44.776255 4014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68c4966fcf3e41b9a0a29a436fda8fb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:44.776468 4011 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68c4966fcf3e41b9a0a29a436fda8fb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:34:44.776544 4014 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68c4966fcf3e41b9a0a29a436fda8fb1 [term 1 LEADER]: Becoming Leader. State: Replica: 68c4966fcf3e41b9a0a29a436fda8fb1, State: Running, Role: LEADER
I20260928 02:34:44.776707 4014 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68c4966fcf3e41b9a0a29a436fda8fb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c4966fcf3e41b9a0a29a436fda8fb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45427 } }
I20260928 02:34:44.777099 4016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68c4966fcf3e41b9a0a29a436fda8fb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68c4966fcf3e41b9a0a29a436fda8fb1. Latest consensus state: current_term: 1 leader_uuid: "68c4966fcf3e41b9a0a29a436fda8fb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c4966fcf3e41b9a0a29a436fda8fb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45427 } } }
I20260928 02:34:44.777100 4015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68c4966fcf3e41b9a0a29a436fda8fb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68c4966fcf3e41b9a0a29a436fda8fb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c4966fcf3e41b9a0a29a436fda8fb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45427 } } }
I20260928 02:34:44.777204 4016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68c4966fcf3e41b9a0a29a436fda8fb1 [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:44.777207 4015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68c4966fcf3e41b9a0a29a436fda8fb1 [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:44.777506 4025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:34:44.778090 4025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:34:44.780179 4025 catalog_manager.cc:1347] Generated new cluster ID: edd271c8c2a4470f8bfe3fb247f3e568
I20260928 02:34:44.780226 4025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:34:44 dist-test-slave-fz47 krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:44.787370 4019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:44.776835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51266 (local address 127.0.98.62:45427)
0928 02:34:44.776994 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:44.777003 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:44.777032 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:34:44.777334 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:44.777342 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:44.777412 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:44.778287 (+ 875us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:44.778366 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:44.779238 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:44.780197 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:44.780759 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:44.780863 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:44.783411 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:44.783415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:44.783416 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:44.783427 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:44.785859 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:44.786210 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:44.786217 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:44.786218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:44.786327 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:44.786496 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:44.786496 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:44.786497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:44.786808 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:44.786958 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:44.787032 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 02:34:44.787163 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":62,"threads_started":1}
I20260928 02:34:44.797031 4025 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:34:44.797781 4025 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:34:44.803189 4025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68c4966fcf3e41b9a0a29a436fda8fb1: Generated new TSK 0
I20260928 02:34:44.803362 4025 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 02:34:44.831910 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45427
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:44.909873 4037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:44.910066 4037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:44.910087 4037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:44.911171 4037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:44.911206 4037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:44.911230 4037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:34:44.912626 4037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:44.912799 4037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:44.912966 4037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:44.917546 4042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:44.917533 4045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:44.917533 4043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:44.917557 4037 server_base.cc:1034] running on GCE node
I20260928 02:34:44.917966 4037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:44.918203 4037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:44.919345 4037 hybrid_clock.cc:648] HybridClock initialized: now 1790562884919331 us; error 33 us; skew 500 ppm
Sep 28 02:34:44 dist-test-slave-fz47 krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:44.921609 4037 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 02:34:44.922012 4037 webserver.cc:460] Webserver started at http://127.0.98.1:33555/ using document root <none> and password file <none>
I20260928 02:34:44.922207 4037 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:44.922252 4037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:44.922367 4037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:44.923336 4037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-0/data/instance:
uuid: "d7b6e44fee3a4188b6773cb18d1e9b91"
format_stamp: "Formatted at 2026-09-28 02:34:44 on dist-test-slave-fz47"
server_key: "6981ad39c123eebae8a9238b7f014879"
server_key_iv: "f1dada0ec4638cbb8eb3794c34e3fa8a"
server_key_version: "encryptionkey@0"
I20260928 02:34:44.923646 4037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d7b6e44fee3a4188b6773cb18d1e9b91"
format_stamp: "Formatted at 2026-09-28 02:34:44 on dist-test-slave-fz47"
server_key: "6981ad39c123eebae8a9238b7f014879"
server_key_iv: "f1dada0ec4638cbb8eb3794c34e3fa8a"
server_key_version: "encryptionkey@0"
I20260928 02:34:44.924782 4037 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:34:44.925441 4052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:44.925589 4037 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:44.925668 4037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "d7b6e44fee3a4188b6773cb18d1e9b91"
format_stamp: "Formatted at 2026-09-28 02:34:44 on dist-test-slave-fz47"
server_key: "6981ad39c123eebae8a9238b7f014879"
server_key_iv: "f1dada0ec4638cbb8eb3794c34e3fa8a"
server_key_version: "encryptionkey@0"
I20260928 02:34:44.925738 4037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:44.935691 4037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:44.937348 4037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:44.937472 4037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:44.937662 4037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:34:44.937932 4037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:44.937964 4037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:44.937991 4037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:44.938014 4037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:44.943181 4037 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36429
I20260928 02:34:44.943239 4165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36429 every 8 connection(s)
I20260928 02:34:44.943506 4037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
I20260928 02:34:44.946061 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 4037
I20260928 02:34:44.946169 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-0/wal/instance
I20260928 02:34:44.946350 392 external_mini_cluster.cc:1428] Setting key 43ab8713eb09c490c28309a1552b6253
Sep 28 02:34:44 dist-test-slave-fz47 krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:44.952421 4019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:44.944126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37585 (local address 127.0.98.62:45427)
0928 02:34:44.944190 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:44.944192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:44.944372 (+ 180us) server_negotiation.cc:407] Connection header received
0928 02:34:44.945230 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:44.945231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:44.945240 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:44.945321 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:44.945904 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:44.946618 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:44.947595 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:44.947822 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:44.947875 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:44.949801 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:44.949803 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:44.949804 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:44.949811 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:44.951219 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:44.951580 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:44.951581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:44.951581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:44.951647 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:44.951837 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:44.951838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:44.951838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:44.951968 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:44.952033 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:44.952243 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 02:34:44.952330 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 02:34:44.952478 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:44.944103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.1:37585)
0928 02:34:44.944250 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:44.944289 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:44.945151 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:44.945373 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:44.945376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:44.945608 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:44.945829 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:44.945835 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:44.946693 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:44.946694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:44.947518 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:44.947520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:44.947888 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:44.947889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:44.947996 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:44.948431 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:44.948439 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:44.949712 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:44.951291 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:44.951296 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:44.951298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:44.951514 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:44.951703 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:44.951703 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:44.951704 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:44.951787 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:44.952022 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:44.952024 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:44.952129 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 02:34:44.952203 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 02:34:44.952312 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":72,"threads_started":1}
I20260928 02:34:44.952929 4166 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45427
I20260928 02:34:44.953022 4166 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:44.953194 4166 heartbeater.cc:511] Master 127.0.98.62:45427 requested a full tablet report, sending...
I20260928 02:34:44.953701 3975 ts_manager.cc:194] Registered new tserver with Master: d7b6e44fee3a4188b6773cb18d1e9b91 (127.0.98.1:36429)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 02:34:44.954284 3975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37585
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 02:34:44.967494 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45427
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:45.044256 4171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:45.044442 4171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:45.044472 4171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:45.045603 4171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:45.045639 4171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:45.045673 4171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:34:45.047081 4171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:45.047305 4171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:45.047494 4171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:45.052114 4179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:45.052052 4176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:45.052057 4177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:45.052301 4171 server_base.cc:1034] running on GCE node
I20260928 02:34:45.052528 4171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:45.052765 4171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:45.053905 4171 hybrid_clock.cc:648] HybridClock initialized: now 1790562885053892 us; error 29 us; skew 500 ppm
Sep 28 02:34:45 dist-test-slave-fz47 krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:45.056123 4171 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 02:34:45.056520 4171 webserver.cc:460] Webserver started at http://127.0.98.2:39419/ using document root <none> and password file <none>
I20260928 02:34:45.056727 4171 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:45.056772 4171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:45.056887 4171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:45.057904 4171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-1/data/instance:
uuid: "7803c6b9ce0e4f18a9824db1fbeb031d"
format_stamp: "Formatted at 2026-09-28 02:34:45 on dist-test-slave-fz47"
server_key: "7fa34deea99871612f299eb21b94fb75"
server_key_iv: "44670e7a774d53f520bceaaa5e7f54d0"
server_key_version: "encryptionkey@0"
I20260928 02:34:45.058209 4171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7803c6b9ce0e4f18a9824db1fbeb031d"
format_stamp: "Formatted at 2026-09-28 02:34:45 on dist-test-slave-fz47"
server_key: "7fa34deea99871612f299eb21b94fb75"
server_key_iv: "44670e7a774d53f520bceaaa5e7f54d0"
server_key_version: "encryptionkey@0"
I20260928 02:34:45.059345 4171 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:34:45.059984 4186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:45.060127 4171 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:45.060209 4171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "7803c6b9ce0e4f18a9824db1fbeb031d"
format_stamp: "Formatted at 2026-09-28 02:34:45 on dist-test-slave-fz47"
server_key: "7fa34deea99871612f299eb21b94fb75"
server_key_iv: "44670e7a774d53f520bceaaa5e7f54d0"
server_key_version: "encryptionkey@0"
I20260928 02:34:45.060268 4171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:45.085676 4171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:45.087409 4171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:45.087541 4171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:45.087759 4171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:34:45.088044 4171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:45.088075 4171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:45.088104 4171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:45.088119 4171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:45.093304 4171 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33419
I20260928 02:34:45.093325 4299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33419 every 8 connection(s)
I20260928 02:34:45.093640 4171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 02:34:45 dist-test-slave-fz47 krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:45.101559 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 4171
I20260928 02:34:45.101675 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-1/wal/instance
I20260928 02:34:45.101877 392 external_mini_cluster.cc:1428] Setting key 558967c483b25b4b0503b49831bed15f
I20260928 02:34:45.101908 4019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:45.094280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34465 (local address 127.0.98.62:45427)
0928 02:34:45.094333 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:45.094335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:45.094532 (+ 197us) server_negotiation.cc:407] Connection header received
0928 02:34:45.095412 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:45.095413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:45.095421 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:45.095474 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:45.096027 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:45.096510 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:45.097501 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:45.097697 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:45.097745 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:45.099572 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:45.099573 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:45.099574 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:45.099580 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:45.100787 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:45.101076 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:45.101077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:45.101077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:45.101123 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:45.101330 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:45.101330 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:45.101331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:45.101426 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:45.101471 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:45.101779 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 02:34:45.101844 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 02:34:45.102217 4302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:45.094256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.2:34465)
0928 02:34:45.094413 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:45.094456 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:45.095333 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:45.095531 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:45.095534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:45.095779 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:45.095968 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:45.095972 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:45.096586 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:45.096587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:45.097435 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:45.097437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:45.097755 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:45.097756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:45.097848 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:45.098261 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:45.098269 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:45.099492 (+ 1223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:45.100843 (+ 1351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:45.100845 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:45.100847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:45.101027 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:45.101168 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:45.101169 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:45.101169 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:45.101275 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:45.101480 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:45.101485 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:45.101641 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 02:34:45.101747 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 02:34:45.101899 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":75,"threads_started":1}
I20260928 02:34:45.102698 4300 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45427
I20260928 02:34:45.102797 4300 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:45.102963 4300 heartbeater.cc:511] Master 127.0.98.62:45427 requested a full tablet report, sending...
I20260928 02:34:45.103317 3975 ts_manager.cc:194] Registered new tserver with Master: 7803c6b9ce0e4f18a9824db1fbeb031d (127.0.98.2:33419)
I20260928 02:34:45.103747 3975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34465
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 02:34:45.123605 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45427
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:45.202319 4305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:45.202504 4305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:45.202535 4305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:45.203738 4305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:45.203780 4305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:45.203814 4305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:34:45.205221 4305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:45.205452 4305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:45.205639 4305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:45.210207 4311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:45.210217 4310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:45.210260 4305 server_base.cc:1034] running on GCE node
W20260928 02:34:45.210376 4313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:45.210614 4305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:45.210837 4305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:45.211982 4305 hybrid_clock.cc:648] HybridClock initialized: now 1790562885211967 us; error 26 us; skew 500 ppm
Sep 28 02:34:45 dist-test-slave-fz47 krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:45.214366 4305 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 02:34:45.214753 4305 webserver.cc:460] Webserver started at http://127.0.98.3:36479/ using document root <none> and password file <none>
I20260928 02:34:45.214948 4305 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:45.214991 4305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:45.215107 4305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:45.216053 4305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-2/data/instance:
uuid: "517db583043d4e0eacb9e9758b4e16ff"
format_stamp: "Formatted at 2026-09-28 02:34:45 on dist-test-slave-fz47"
server_key: "fef985772bcf0289e3fc761f57c9cf9f"
server_key_iv: "6da022734396ba529609079d233439b7"
server_key_version: "encryptionkey@0"
I20260928 02:34:45.216358 4305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-2/wal/instance:
uuid: "517db583043d4e0eacb9e9758b4e16ff"
format_stamp: "Formatted at 2026-09-28 02:34:45 on dist-test-slave-fz47"
server_key: "fef985772bcf0289e3fc761f57c9cf9f"
server_key_iv: "6da022734396ba529609079d233439b7"
server_key_version: "encryptionkey@0"
I20260928 02:34:45.217511 4305 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:34:45.218123 4320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:45.218258 4305 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:34:45.218327 4305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "517db583043d4e0eacb9e9758b4e16ff"
format_stamp: "Formatted at 2026-09-28 02:34:45 on dist-test-slave-fz47"
server_key: "fef985772bcf0289e3fc761f57c9cf9f"
server_key_iv: "6da022734396ba529609079d233439b7"
server_key_version: "encryptionkey@0"
I20260928 02:34:45.218384 4305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:45.248448 4305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:45.251037 4305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:45.251215 4305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:45.251466 4305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:34:45.251775 4305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:45.251806 4305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:45.251835 4305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:45.251860 4305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:45.258102 4305 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41227
I20260928 02:34:45.258165 4433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41227 every 8 connection(s)
I20260928 02:34:45.258478 4305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 02:34:45 dist-test-slave-fz47 krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:45.267498 4019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:45.259305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37257 (local address 127.0.98.62:45427)
0928 02:34:45.259379 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:45.259381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:45.259499 (+ 118us) server_negotiation.cc:407] Connection header received
0928 02:34:45.260454 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:45.260455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:45.260467 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:45.260564 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:45.261180 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:45.261712 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:45.262749 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:45.262931 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:45.262984 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:45.264998 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:45.264999 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:45.265000 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:45.265006 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:45.266263 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:45.266668 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:45.266669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:45.266669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:45.266715 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:45.266990 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:45.266990 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:45.266991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:45.267084 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:45.267125 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:45.267376 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 02:34:45.267433 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:34:45.267694 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:45.259230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45427 (local address 127.0.98.3:37257)
0928 02:34:45.259381 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:45.259424 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:45.260331 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:45.260631 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:45.260634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:45.260897 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:45.261089 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:45.261093 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:45.261790 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:45.261791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:45.262673 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:45.262676 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:45.262990 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:45.262991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:45.263143 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:45.263609 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:45.263619 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:45.264918 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:45.266365 (+ 1447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:45.266372 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:45.266374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:45.266608 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:45.266787 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:45.266788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:45.266789 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:45.266906 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:45.267136 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:45.267141 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:45.267260 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 02:34:45.267350 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 02:34:45.267481 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260928 02:34:45.268092 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 4305
I20260928 02:34:45.268200 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790562862804661-392-0/minicluster-data/ts-2/wal/instance
I20260928 02:34:45.268261 4434 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45427
I20260928 02:34:45.268368 4434 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:45.268389 392 external_mini_cluster.cc:1428] Setting key d4d3af5d01e528a3c9d65c357de3e5b5
I20260928 02:34:45.268559 4434 heartbeater.cc:511] Master 127.0.98.62:45427 requested a full tablet report, sending...
I20260928 02:34:45.268957 3975 ts_manager.cc:194] Registered new tserver with Master: 517db583043d4e0eacb9e9758b4e16ff (127.0.98.3:41227)
I20260928 02:34:45.269304 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:34:45.269389 3975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37257
Sep 28 02:34:45 dist-test-slave-fz47 krb5kdc[3930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:34:45 dist-test-slave-fz47 krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 02:34:45.291117 4441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:45.283809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39204 (local address 127.0.98.1:36429)
0928 02:34:45.283971 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:45.283977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:45.283987 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:34:45.284012 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:45.284014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:45.284028 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:45.284089 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:45.284671 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:45.285470 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:45.286180 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:45.286497 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:45.286556 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:45.288608 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:45.288618 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:45.288618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:45.288625 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:45.290046 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:45.290372 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:45.290375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:45.290376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:45.290446 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:45.290606 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:45.290607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:45.290607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:45.290748 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:45.290889 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:45.290935 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 02:34:45.291011 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":70,"threads_started":1}
Sep 28 02:34:45 dist-test-slave-fz47 krb5kdc[3930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:45.298188 4019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:45.292361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51276 (local address 127.0.98.62:45427)
0928 02:34:45.292416 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:45.292418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:45.292428 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:34:45.292460 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:45.292461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:45.292469 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:45.292507 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:45.292921 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:45.293454 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:45.294097 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:45.294291 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:45.294341 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:45.296243 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:45.296245 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:45.296245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:45.296252 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:45.297340 (+ 1088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:45.297614 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:45.297615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:45.297615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:45.297657 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:45.297795 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:45.297795 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:45.297795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:45.297919 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:45.297987 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:45.298054 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 02:34:45.298125 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 02:34:45.298394 3975 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51276
I20260928 02:34:45.298698 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 4037
I20260928 02:34:45.302757 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 4171
I20260928 02:34:45.306751 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 4305
I20260928 02:34:45.310480 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 3943
2026-09-28T02:34:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1695 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:34:45 dist-test-slave-fz47 krb5kdc[4446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:34:45 dist-test-slave-fz47 krb5kdc[4446](info): set up 2 sockets
Sep 28 02:34:45 dist-test-slave-fz47 krb5kdc[4446](info): commencing operation
krb5kdc: starting...
W20260928 02:34:46.260922 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:34:46 dist-test-slave-fz47 krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:34:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:34:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 02:34:46.370229 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34761
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:46.445855 4459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:46.446040 4459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:46.446060 4459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:46.447129 4459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:34:46.447161 4459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:46.447173 4459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:46.447184 4459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:34:46.447194 4459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:34:46.448567 4459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:46.448766 4459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:46.448974 4459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:46.453243 4467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:46.453231 4464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:46.453384 4465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:46.453485 4459 server_base.cc:1034] running on GCE node
I20260928 02:34:46.453742 4459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:46.454012 4459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:46.455148 4459 hybrid_clock.cc:648] HybridClock initialized: now 1790562886455138 us; error 30 us; skew 500 ppm
Sep 28 02:34:46 dist-test-slave-fz47 krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:46.457304 4459 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:34:46.457580 4459 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:46.457628 4459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:46.457741 4459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:46.458683 4459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "6328d0d4c309491e9c9b3e8c20ebe49a"
format_stamp: "Formatted at 2026-09-28 02:34:46 on dist-test-slave-fz47"
server_key: "d4f09da562fd11955a8d7ab7156be596"
server_key_iv: "d6f8f198013fc9914462355b944463a2"
server_key_version: "encryptionkey@0"
I20260928 02:34:46.458998 4459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "6328d0d4c309491e9c9b3e8c20ebe49a"
format_stamp: "Formatted at 2026-09-28 02:34:46 on dist-test-slave-fz47"
server_key: "d4f09da562fd11955a8d7ab7156be596"
server_key_iv: "d6f8f198013fc9914462355b944463a2"
server_key_version: "encryptionkey@0"
I20260928 02:34:46.460088 4459 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:34:46.460664 4473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:46.460806 4459 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:46.460882 4459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "6328d0d4c309491e9c9b3e8c20ebe49a"
format_stamp: "Formatted at 2026-09-28 02:34:46 on dist-test-slave-fz47"
server_key: "d4f09da562fd11955a8d7ab7156be596"
server_key_iv: "d6f8f198013fc9914462355b944463a2"
server_key_version: "encryptionkey@0"
I20260928 02:34:46.460953 4459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:46.478860 4459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:46.480616 4459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:46.480738 4459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:46.484280 4459 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34761
I20260928 02:34:46.484309 4525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34761 every 8 connection(s)
I20260928 02:34:46.484565 4459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:34:46.485435 4526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:46.487535 4526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6328d0d4c309491e9c9b3e8c20ebe49a: Bootstrap starting.
I20260928 02:34:46.488075 4526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6328d0d4c309491e9c9b3e8c20ebe49a: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:46.488327 4526 log.cc:824] T 00000000000000000000000000000000 P 6328d0d4c309491e9c9b3e8c20ebe49a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:46.488889 4526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6328d0d4c309491e9c9b3e8c20ebe49a: No bootstrap required, opened a new log
I20260928 02:34:46.490408 4526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6328d0d4c309491e9c9b3e8c20ebe49a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6328d0d4c309491e9c9b3e8c20ebe49a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34761 } }
I20260928 02:34:46.490540 4526 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6328d0d4c309491e9c9b3e8c20ebe49a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:46.490562 4526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6328d0d4c309491e9c9b3e8c20ebe49a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6328d0d4c309491e9c9b3e8c20ebe49a, State: Initialized, Role: FOLLOWER
I20260928 02:34:46.490698 4526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6328d0d4c309491e9c9b3e8c20ebe49a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6328d0d4c309491e9c9b3e8c20ebe49a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34761 } }
I20260928 02:34:46.490759 4526 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6328d0d4c309491e9c9b3e8c20ebe49a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:34:46.490778 4526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6328d0d4c309491e9c9b3e8c20ebe49a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:34:46.490804 4526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6328d0d4c309491e9c9b3e8c20ebe49a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:46.491559 4526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6328d0d4c309491e9c9b3e8c20ebe49a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6328d0d4c309491e9c9b3e8c20ebe49a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34761 } }
I20260928 02:34:46.491637 4526 leader_election.cc:302] T 00000000000000000000000000000000 P 6328d0d4c309491e9c9b3e8c20ebe49a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6328d0d4c309491e9c9b3e8c20ebe49a; no voters:
I20260928 02:34:46.491782 4526 leader_election.cc:288] T 00000000000000000000000000000000 P 6328d0d4c309491e9c9b3e8c20ebe49a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:34:46.491832 4529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6328d0d4c309491e9c9b3e8c20ebe49a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:46.491988 4529 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6328d0d4c309491e9c9b3e8c20ebe49a [term 1 LEADER]: Becoming Leader. State: Replica: 6328d0d4c309491e9c9b3e8c20ebe49a, State: Running, Role: LEADER
I20260928 02:34:46.491997 4526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6328d0d4c309491e9c9b3e8c20ebe49a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:34:46.492136 4529 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6328d0d4c309491e9c9b3e8c20ebe49a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6328d0d4c309491e9c9b3e8c20ebe49a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34761 } }
I20260928 02:34:46.492453 4531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6328d0d4c309491e9c9b3e8c20ebe49a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6328d0d4c309491e9c9b3e8c20ebe49a. Latest consensus state: current_term: 1 leader_uuid: "6328d0d4c309491e9c9b3e8c20ebe49a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6328d0d4c309491e9c9b3e8c20ebe49a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34761 } } }
I20260928 02:34:46.492553 4531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6328d0d4c309491e9c9b3e8c20ebe49a [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:46.492455 4530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6328d0d4c309491e9c9b3e8c20ebe49a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6328d0d4c309491e9c9b3e8c20ebe49a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6328d0d4c309491e9c9b3e8c20ebe49a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34761 } } }
I20260928 02:34:46.492713 4530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6328d0d4c309491e9c9b3e8c20ebe49a [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:46.493036 4537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:34:46.493479 4537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:34:46.494130 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 4459
I20260928 02:34:46.494231 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:34:46.494390 392 external_mini_cluster.cc:1428] Setting key fedab78f48d73bbf70a7509d3f41cfbc
I20260928 02:34:46.495329 4537 catalog_manager.cc:1347] Generated new cluster ID: c433042092a6444e9bf7e3fb7e38d899
I20260928 02:34:46.495370 4537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:34:46 dist-test-slave-fz47 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:46.505880 4549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:46.494786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33384 (local address 127.0.98.62:34761)
0928 02:34:46.495406 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:46.495414 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:46.495444 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:34:46.495550 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:46.495557 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:46.495608 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:46.496491 (+ 883us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:46.496580 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:46.498022 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:46.499074 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:46.499700 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:46.500087 (+ 387us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:46.502325 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:46.502329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:46.502330 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:46.502340 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:46.504127 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:46.504472 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:46.504475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:46.504475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:46.504573 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:46.504734 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:46.504735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:46.504735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:46.505018 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:46.505454 (+ 436us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:46.505530 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 02:34:46.505662 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":53,"threads_started":1}
I20260928 02:34:46.519124 4537 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:34:46.519642 4537 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:34:46.528529 4537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6328d0d4c309491e9c9b3e8c20ebe49a: Generated new TSK 0
I20260928 02:34:46.528694 4537 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 02:34:46.556219 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34761
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:46.634368 4552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:46.634577 4552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:46.634611 4552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:46.635779 4552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:46.635823 4552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:46.635856 4552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:34:46.637334 4552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:46.637562 4552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:46.637758 4552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:46.642299 4557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:46.642445 4558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:46.642530 4552 server_base.cc:1034] running on GCE node
W20260928 02:34:46.642299 4560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:46.642781 4552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:46.643002 4552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:46.644146 4552 hybrid_clock.cc:648] HybridClock initialized: now 1790562886644126 us; error 36 us; skew 500 ppm
Sep 28 02:34:46 dist-test-slave-fz47 krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:46.646584 4552 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 02:34:46.646863 4552 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:46.646905 4552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:46.646989 4552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:46.647972 4552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-0/data/instance:
uuid: "bcaa8422059f40e0bd40736a5db52dcf"
format_stamp: "Formatted at 2026-09-28 02:34:46 on dist-test-slave-fz47"
server_key: "134047675684d987e4820a4fc34c7773"
server_key_iv: "007c47d8909bc7574b9e87eb6eda540d"
server_key_version: "encryptionkey@0"
I20260928 02:34:46.648265 4552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bcaa8422059f40e0bd40736a5db52dcf"
format_stamp: "Formatted at 2026-09-28 02:34:46 on dist-test-slave-fz47"
server_key: "134047675684d987e4820a4fc34c7773"
server_key_iv: "007c47d8909bc7574b9e87eb6eda540d"
server_key_version: "encryptionkey@0"
I20260928 02:34:46.649483 4552 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:34:46.650224 4566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:46.650424 4552 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:34:46.650501 4552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "bcaa8422059f40e0bd40736a5db52dcf"
format_stamp: "Formatted at 2026-09-28 02:34:46 on dist-test-slave-fz47"
server_key: "134047675684d987e4820a4fc34c7773"
server_key_iv: "007c47d8909bc7574b9e87eb6eda540d"
server_key_version: "encryptionkey@0"
I20260928 02:34:46.650565 4552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:46.658762 4552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:46.660467 4552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:46.660578 4552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:46.660786 4552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:34:46.661079 4552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:46.661110 4552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:46.661130 4552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:46.661152 4552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:46.666478 4552 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37939
I20260928 02:34:46.666545 4679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37939 every 8 connection(s)
I20260928 02:34:46.666755 4552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
I20260928 02:34:46.670624 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 4552
I20260928 02:34:46.670717 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-0/wal/instance
I20260928 02:34:46.670902 392 external_mini_cluster.cc:1428] Setting key 396a6d4d7caef3adcea82065e9665d59
Sep 28 02:34:46 dist-test-slave-fz47 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:46.675376 4549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:46.667413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47823 (local address 127.0.98.62:34761)
0928 02:34:46.667460 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:46.667462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:46.667645 (+ 183us) server_negotiation.cc:407] Connection header received
0928 02:34:46.668124 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:46.668125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:46.668138 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:46.668172 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:46.669007 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:46.669553 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:46.670488 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:46.670793 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:46.670860 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:46.672885 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:46.672887 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:46.672887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:46.672896 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:46.674143 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:46.674533 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:46.674534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:46.674535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:46.674585 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:46.674791 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:46.674792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:46.674793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:46.674895 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:46.674948 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:46.675207 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 02:34:46.675290 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 02:34:46.675529 4682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:46.667391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34761 (local address 127.0.98.1:47823)
0928 02:34:46.667532 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:46.667567 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:46.668031 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:46.668234 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:46.668237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:46.668480 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:46.668942 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:46.668948 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:46.669622 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:46.669622 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:46.670414 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:46.670416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:46.670877 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:46.670878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:46.670982 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:46.671468 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:46.671477 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:46.672801 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:46.674214 (+ 1413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:46.674220 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:46.674222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:46.674462 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:46.674636 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:46.674637 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:46.674637 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:46.674739 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:46.674961 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:46.674966 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:46.675094 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 02:34:46.675179 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 02:34:46.675328 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
I20260928 02:34:46.675940 4680 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34761
I20260928 02:34:46.676031 4680 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:46.676168 4680 heartbeater.cc:511] Master 127.0.98.62:34761 requested a full tablet report, sending...
I20260928 02:34:46.676700 4490 ts_manager.cc:194] Registered new tserver with Master: bcaa8422059f40e0bd40736a5db52dcf (127.0.98.1:37939)
I20260928 02:34:46.677618 4490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47823
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 02:34:46.693830 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34761
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:46.770985 4685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:46.771165 4685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:46.771188 4685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:46.772312 4685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:46.772351 4685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:46.772375 4685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:34:46.773837 4685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:46.774040 4685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:46.774236 4685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:46.778687 4693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:46.778750 4691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:46.778801 4690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:46.778965 4685 server_base.cc:1034] running on GCE node
I20260928 02:34:46.779192 4685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:46.779415 4685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:46.780582 4685 hybrid_clock.cc:648] HybridClock initialized: now 1790562886780567 us; error 27 us; skew 500 ppm
Sep 28 02:34:46 dist-test-slave-fz47 krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:46.782946 4685 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 02:34:46.783249 4685 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:46.783298 4685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:46.783413 4685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:46.784376 4685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-1/data/instance:
uuid: "fb96c2270b934e9ab2990eae86476aa3"
format_stamp: "Formatted at 2026-09-28 02:34:46 on dist-test-slave-fz47"
server_key: "7bf9b12f0d411253dc7e782e5ecf85e5"
server_key_iv: "4b7e8e5588ee613536b8ba0f70d5d9fc"
server_key_version: "encryptionkey@0"
I20260928 02:34:46.784678 4685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fb96c2270b934e9ab2990eae86476aa3"
format_stamp: "Formatted at 2026-09-28 02:34:46 on dist-test-slave-fz47"
server_key: "7bf9b12f0d411253dc7e782e5ecf85e5"
server_key_iv: "4b7e8e5588ee613536b8ba0f70d5d9fc"
server_key_version: "encryptionkey@0"
I20260928 02:34:46.785847 4685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:34:46.786552 4699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:46.786720 4685 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:34:46.786795 4685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "fb96c2270b934e9ab2990eae86476aa3"
format_stamp: "Formatted at 2026-09-28 02:34:46 on dist-test-slave-fz47"
server_key: "7bf9b12f0d411253dc7e782e5ecf85e5"
server_key_iv: "4b7e8e5588ee613536b8ba0f70d5d9fc"
server_key_version: "encryptionkey@0"
I20260928 02:34:46.786852 4685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:46.808037 4685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:46.809844 4685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:46.809978 4685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:46.810168 4685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:34:46.810432 4685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:46.810464 4685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:46.810491 4685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:46.810515 4685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:46.815601 4685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43289
I20260928 02:34:46.815657 4812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43289 every 8 connection(s)
I20260928 02:34:46.815867 4685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
I20260928 02:34:46.817886 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 4685
I20260928 02:34:46.817996 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-1/wal/instance
I20260928 02:34:46.818166 392 external_mini_cluster.cc:1428] Setting key 51d39b05276b3879f654520474e5afcf
Sep 28 02:34:46 dist-test-slave-fz47 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:46.824540 4549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:46.816488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39879 (local address 127.0.98.62:34761)
0928 02:34:46.816561 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:46.816564 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:46.816699 (+ 135us) server_negotiation.cc:407] Connection header received
0928 02:34:46.817147 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:46.817148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:46.817157 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:46.817203 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:46.818159 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:46.818776 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:46.819734 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:46.819985 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:46.820038 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:46.822153 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:46.822155 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:46.822155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:46.822162 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:46.823382 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:46.823735 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:46.823736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:46.823737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:46.823785 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:46.823987 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:46.823988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:46.823989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:46.824088 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:46.824141 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:46.824380 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 02:34:46.824454 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:34:46.824702 4815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:46.816453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34761 (local address 127.0.98.2:39879)
0928 02:34:46.816587 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:46.816620 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:46.817070 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:46.817290 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:46.817295 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:46.817548 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:46.818072 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:46.818078 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:46.818873 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:46.818874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:46.819651 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:46.819653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:46.820056 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:46.820057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:46.820197 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:46.820664 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:46.820673 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:46.822073 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:46.823466 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:46.823472 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:46.823474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:46.823672 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:46.823836 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:46.823837 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:46.823837 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:46.823931 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:46.824139 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:46.824145 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:46.824269 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 02:34:46.824351 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 02:34:46.824503 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":74,"threads_started":1}
I20260928 02:34:46.825204 4813 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34761
I20260928 02:34:46.825320 4813 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:46.825474 4813 heartbeater.cc:511] Master 127.0.98.62:34761 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 02:34:46.825850 4490 ts_manager.cc:194] Registered new tserver with Master: fb96c2270b934e9ab2990eae86476aa3 (127.0.98.2:43289)
I20260928 02:34:46.826193 4490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 02:34:46.839892 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34761
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:46.915731 4818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:46.915922 4818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:46.915946 4818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:46.917079 4818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:46.917119 4818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:46.917150 4818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:34:46.918596 4818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:46.918764 4818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:46.918924 4818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:46.923430 4826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:46.923504 4818 server_base.cc:1034] running on GCE node
W20260928 02:34:46.923585 4823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:46.923623 4824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:46.923928 4818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:46.924146 4818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:46.925290 4818 hybrid_clock.cc:648] HybridClock initialized: now 1790562886925275 us; error 32 us; skew 500 ppm
Sep 28 02:34:46 dist-test-slave-fz47 krb5kdc[4446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:46.931290 4818 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 02:34:46.931571 4818 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:46.931612 4818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:46.931699 4818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:46.932684 4818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-2/data/instance:
uuid: "ee3a28f0c7c84c31957e05fc25e4b349"
format_stamp: "Formatted at 2026-09-28 02:34:46 on dist-test-slave-fz47"
server_key: "20e3df3e97c9ace2175d9a5410a0290b"
server_key_iv: "7dc2e6bcb82a25b90816a22945a05509"
server_key_version: "encryptionkey@0"
I20260928 02:34:46.932986 4818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ee3a28f0c7c84c31957e05fc25e4b349"
format_stamp: "Formatted at 2026-09-28 02:34:46 on dist-test-slave-fz47"
server_key: "20e3df3e97c9ace2175d9a5410a0290b"
server_key_iv: "7dc2e6bcb82a25b90816a22945a05509"
server_key_version: "encryptionkey@0"
I20260928 02:34:46.934130 4818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 02:34:46.934760 4832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:46.934902 4818 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:34:46.934981 4818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "ee3a28f0c7c84c31957e05fc25e4b349"
format_stamp: "Formatted at 2026-09-28 02:34:46 on dist-test-slave-fz47"
server_key: "20e3df3e97c9ace2175d9a5410a0290b"
server_key_iv: "7dc2e6bcb82a25b90816a22945a05509"
server_key_version: "encryptionkey@0"
I20260928 02:34:46.935050 4818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:46.951920 4818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:46.953572 4818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:46.953694 4818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:46.953881 4818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:34:46.954131 4818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:46.954165 4818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:46.954185 4818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:46.954200 4818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:46.959319 4818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34901
I20260928 02:34:46.959352 4945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34901 every 8 connection(s)
I20260928 02:34:46.959578 4818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
I20260928 02:34:46.963874 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 4818
I20260928 02:34:46.963963 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-2/wal/instance
I20260928 02:34:46.964134 392 external_mini_cluster.cc:1428] Setting key 0ac9f514bde386c83d77b07e3a8a0321
Sep 28 02:34:46 dist-test-slave-fz47 krb5kdc[4446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:46.967608 4549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:46.960173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53953 (local address 127.0.98.62:34761)
0928 02:34:46.960214 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:46.960216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:46.960396 (+ 180us) server_negotiation.cc:407] Connection header received
0928 02:34:46.960817 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:46.960818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:46.960842 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:46.960894 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:46.961725 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:46.962210 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:46.963138 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:46.963342 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:46.963392 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:46.965517 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:46.965518 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:46.965518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:46.965524 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:46.966522 (+ 998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:46.966862 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:46.966863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:46.966864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:46.966921 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:46.967104 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:46.967105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:46.967105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:46.967218 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:46.967262 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:46.967486 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 02:34:46.967547 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:34:46.967754 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:46.960139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34761 (local address 127.0.98.3:53953)
0928 02:34:46.960286 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:46.960325 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:46.960748 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:46.960946 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:46.960949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:46.961181 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:46.961649 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:46.961654 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:46.962275 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:46.962275 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:46.963074 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:46.963077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:46.963398 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:46.963398 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:46.963487 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:46.963898 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:46.963906 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:46.965433 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:46.966581 (+ 1148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:46.966586 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:46.966587 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:46.966774 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:46.966980 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:46.966981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:46.966981 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:46.967062 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:46.967265 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:46.967269 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:46.967389 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 02:34:46.967466 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 02:34:46.967572 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
I20260928 02:34:46.968170 4946 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34761
I20260928 02:34:46.968250 4946 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:46.968380 4946 heartbeater.cc:511] Master 127.0.98.62:34761 requested a full tablet report, sending...
I20260928 02:34:46.968703 4490 ts_manager.cc:194] Registered new tserver with Master: ee3a28f0c7c84c31957e05fc25e4b349 (127.0.98.3:34901)
I20260928 02:34:46.969051 4490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53953
I20260928 02:34:46.975694 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:34:46.981878 4549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:46.976867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33398 (local address 127.0.98.62:34761)
0928 02:34:46.976909 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:46.976910 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:46.976945 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:34:46.976991 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:46.976992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:46.976999 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:46.977038 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:46.977570 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:46.978019 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:46.978644 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:46.978811 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:46.978857 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:46.979846 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:46.979847 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:46.979847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:46.979853 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:46.980981 (+ 1128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:46.981279 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:46.981281 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:46.981281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:46.981342 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:46.981496 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:46.981497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:46.981497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:46.981616 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:46.981681 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:46.981759 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 02:34:46.981814 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:34:46.983096 4490 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:34:46.984616 4490 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:34:46.990254 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:46.986487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34901 (local address 127.0.0.1:51430)
0928 02:34:46.986633 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:46.986670 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:46.986733 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:46.987128 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:46.987131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:46.987133 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:46.987192 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:46.987193 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:46.988092 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:46.988093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:46.989204 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:46.989207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:46.989993 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:46.989994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:46.990066 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:46.990080 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 02:34:46.990137 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 02:34:46.990184 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":53,"threads_started":1}
I20260928 02:34:46.990265 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:46.986840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51430 (local address 127.0.98.3:34901)
0928 02:34:46.986991 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:46.986995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:46.987003 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:34:46.987031 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:46.987032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:46.987044 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:46.987108 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:46.987321 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:46.988009 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:46.989327 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:46.989922 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:46.989980 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:46.990036 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:46.990156 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 02:34:46.990193 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":93,"threads_started":1}
I20260928 02:34:46.990922 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:46.986499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43289 (local address 127.0.0.1:52100)
0928 02:34:46.986823 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:46.986831 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:46.986884 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:46.987159 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:46.987160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:46.987161 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:46.987210 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:46.987211 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:46.988521 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:46.988522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:46.989590 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:46.989592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:46.990713 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:46.990714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:46.990767 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:46.990771 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:34:46.990819 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 02:34:46.990857 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":3,"thread_start_us":75,"threads_started":1}
I20260928 02:34:46.990947 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:46.986707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52100 (local address 127.0.98.2:43289)
0928 02:34:46.986997 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:46.987001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:46.987010 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:34:46.987048 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:46.987051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:46.987065 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:46.987276 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:46.987348 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:46.988427 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:46.989664 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:46.990662 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:46.990715 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:46.990768 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:46.990832 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 02:34:46.990874 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":72,"threads_started":1}
I20260928 02:34:46.990955 4880 tablet_service.cc:1467] Processing CreateTablet for tablet bbdae08ee279472e840b058663752c17 (DEFAULT_TABLE table=test-table [id=a9c6a422afa74067b6008d40e04eb511]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:46.991307 4880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbdae08ee279472e840b058663752c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:46.991711 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:46.986487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37939 (local address 127.0.0.1:60556)
0928 02:34:46.986707 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:46.986716 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:46.986766 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:46.987231 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:46.987231 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:46.987233 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:46.987266 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:46.987268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:46.989469 (+ 2201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:46.989470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:46.990577 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:46.990580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:46.991506 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:46.991507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:46.991566 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:46.991569 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:34:46.991613 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 02:34:46.991648 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":2,"thread_start_us":51,"threads_started":1}
I20260928 02:34:46.991797 4747 tablet_service.cc:1467] Processing CreateTablet for tablet bbdae08ee279472e840b058663752c17 (DEFAULT_TABLE table=test-table [id=a9c6a422afa74067b6008d40e04eb511]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:46.991909 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:46.986659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60556 (local address 127.0.98.1:37939)
0928 02:34:46.987032 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:46.987036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:46.987045 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:34:46.987077 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:46.987079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:46.987093 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:46.987157 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:46.987333 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:46.988427 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:46.990656 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:46.991307 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:46.991404 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:46.991460 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:46.991801 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 02:34:46.991839 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":87,"threads_started":1}
I20260928 02:34:46.992086 4747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbdae08ee279472e840b058663752c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:46.992555 4614 tablet_service.cc:1467] Processing CreateTablet for tablet bbdae08ee279472e840b058663752c17 (DEFAULT_TABLE table=test-table [id=a9c6a422afa74067b6008d40e04eb511]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:46.992831 4614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbdae08ee279472e840b058663752c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:46.993969 4960 tablet_bootstrap.cc:492] T bbdae08ee279472e840b058663752c17 P ee3a28f0c7c84c31957e05fc25e4b349: Bootstrap starting.
I20260928 02:34:46.994565 4960 tablet_bootstrap.cc:654] T bbdae08ee279472e840b058663752c17 P ee3a28f0c7c84c31957e05fc25e4b349: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:46.995012 4960 log.cc:824] T bbdae08ee279472e840b058663752c17 P ee3a28f0c7c84c31957e05fc25e4b349: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:46.995186 4961 tablet_bootstrap.cc:492] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3: Bootstrap starting.
I20260928 02:34:46.995657 4961 tablet_bootstrap.cc:654] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:46.995772 4963 tablet_bootstrap.cc:492] T bbdae08ee279472e840b058663752c17 P bcaa8422059f40e0bd40736a5db52dcf: Bootstrap starting.
I20260928 02:34:46.996119 4961 log.cc:824] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:46.996368 4963 tablet_bootstrap.cc:654] T bbdae08ee279472e840b058663752c17 P bcaa8422059f40e0bd40736a5db52dcf: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:46.996802 4961 tablet_bootstrap.cc:492] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3: No bootstrap required, opened a new log
I20260928 02:34:46.996814 4960 tablet_bootstrap.cc:492] T bbdae08ee279472e840b058663752c17 P ee3a28f0c7c84c31957e05fc25e4b349: No bootstrap required, opened a new log
I20260928 02:34:46.996824 4963 log.cc:824] T bbdae08ee279472e840b058663752c17 P bcaa8422059f40e0bd40736a5db52dcf: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:46.996865 4960 ts_tablet_manager.cc:1397] T bbdae08ee279472e840b058663752c17 P ee3a28f0c7c84c31957e05fc25e4b349: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 02:34:46.996865 4961 ts_tablet_manager.cc:1397] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:34:46.997447 4963 tablet_bootstrap.cc:492] T bbdae08ee279472e840b058663752c17 P bcaa8422059f40e0bd40736a5db52dcf: No bootstrap required, opened a new log
I20260928 02:34:46.997500 4963 ts_tablet_manager.cc:1397] T bbdae08ee279472e840b058663752c17 P bcaa8422059f40e0bd40736a5db52dcf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:34:46.998733 4960 raft_consensus.cc:348] T bbdae08ee279472e840b058663752c17 P ee3a28f0c7c84c31957e05fc25e4b349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee3a28f0c7c84c31957e05fc25e4b349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34901 } } peers { permanent_uuid: "bcaa8422059f40e0bd40736a5db52dcf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37939 } } peers { permanent_uuid: "fb96c2270b934e9ab2990eae86476aa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } }
I20260928 02:34:46.998733 4961 raft_consensus.cc:348] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee3a28f0c7c84c31957e05fc25e4b349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34901 } } peers { permanent_uuid: "bcaa8422059f40e0bd40736a5db52dcf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37939 } } peers { permanent_uuid: "fb96c2270b934e9ab2990eae86476aa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } }
I20260928 02:34:46.998788 4963 raft_consensus.cc:348] T bbdae08ee279472e840b058663752c17 P bcaa8422059f40e0bd40736a5db52dcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee3a28f0c7c84c31957e05fc25e4b349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34901 } } peers { permanent_uuid: "bcaa8422059f40e0bd40736a5db52dcf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37939 } } peers { permanent_uuid: "fb96c2270b934e9ab2990eae86476aa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } }
I20260928 02:34:46.998901 4963 raft_consensus.cc:374] T bbdae08ee279472e840b058663752c17 P bcaa8422059f40e0bd40736a5db52dcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:46.998901 4961 raft_consensus.cc:374] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:46.998901 4960 raft_consensus.cc:374] T bbdae08ee279472e840b058663752c17 P ee3a28f0c7c84c31957e05fc25e4b349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:46.998931 4963 raft_consensus.cc:729] T bbdae08ee279472e840b058663752c17 P bcaa8422059f40e0bd40736a5db52dcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcaa8422059f40e0bd40736a5db52dcf, State: Initialized, Role: FOLLOWER
I20260928 02:34:46.998937 4960 raft_consensus.cc:729] T bbdae08ee279472e840b058663752c17 P ee3a28f0c7c84c31957e05fc25e4b349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee3a28f0c7c84c31957e05fc25e4b349, State: Initialized, Role: FOLLOWER
I20260928 02:34:46.998937 4961 raft_consensus.cc:729] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb96c2270b934e9ab2990eae86476aa3, State: Initialized, Role: FOLLOWER
I20260928 02:34:46.999070 4963 consensus_queue.cc:260] T bbdae08ee279472e840b058663752c17 P bcaa8422059f40e0bd40736a5db52dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee3a28f0c7c84c31957e05fc25e4b349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34901 } } peers { permanent_uuid: "bcaa8422059f40e0bd40736a5db52dcf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37939 } } peers { permanent_uuid: "fb96c2270b934e9ab2990eae86476aa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } }
I20260928 02:34:46.999081 4960 consensus_queue.cc:260] T bbdae08ee279472e840b058663752c17 P ee3a28f0c7c84c31957e05fc25e4b349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee3a28f0c7c84c31957e05fc25e4b349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34901 } } peers { permanent_uuid: "bcaa8422059f40e0bd40736a5db52dcf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37939 } } peers { permanent_uuid: "fb96c2270b934e9ab2990eae86476aa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } }
I20260928 02:34:46.999109 4961 consensus_queue.cc:260] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee3a28f0c7c84c31957e05fc25e4b349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34901 } } peers { permanent_uuid: "bcaa8422059f40e0bd40736a5db52dcf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37939 } } peers { permanent_uuid: "fb96c2270b934e9ab2990eae86476aa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } }
I20260928 02:34:46.999330 4960 ts_tablet_manager.cc:1428] T bbdae08ee279472e840b058663752c17 P ee3a28f0c7c84c31957e05fc25e4b349: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:34:46.999330 4961 ts_tablet_manager.cc:1428] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:34:46.999331 4963 ts_tablet_manager.cc:1428] T bbdae08ee279472e840b058663752c17 P bcaa8422059f40e0bd40736a5db52dcf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:34:46.999469 4813 heartbeater.cc:503] Master 127.0.98.62:34761 was elected leader, sending a full tablet report...
I20260928 02:34:46.999477 4680 heartbeater.cc:503] Master 127.0.98.62:34761 was elected leader, sending a full tablet report...
I20260928 02:34:46.999776 4946 heartbeater.cc:503] Master 127.0.98.62:34761 was elected leader, sending a full tablet report...
W20260928 02:34:47.066339 4814 tablet.cc:2369] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:34:47.167456 4681 tablet.cc:2369] T bbdae08ee279472e840b058663752c17 P bcaa8422059f40e0bd40736a5db52dcf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:34:47.210094 4947 tablet.cc:2369] T bbdae08ee279472e840b058663752c17 P ee3a28f0c7c84c31957e05fc25e4b349: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:34:47.342003 4968 raft_consensus.cc:482] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:34:47.342118 4968 raft_consensus.cc:504] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee3a28f0c7c84c31957e05fc25e4b349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34901 } } peers { permanent_uuid: "bcaa8422059f40e0bd40736a5db52dcf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37939 } } peers { permanent_uuid: "fb96c2270b934e9ab2990eae86476aa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } }
I20260928 02:34:47.342406 4968 leader_election.cc:288] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee3a28f0c7c84c31957e05fc25e4b349 (127.0.98.3:34901), bcaa8422059f40e0bd40736a5db52dcf (127.0.98.1:37939)
I20260928 02:34:47.345124 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:47.342719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49977 (local address 127.0.98.3:34901)
0928 02:34:47.342780 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:47.342782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:47.342791 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:34:47.342859 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:47.342860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:47.342872 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:47.342923 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:47.343058 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:47.343535 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:47.344477 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:47.344827 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:47.344886 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:47.344895 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:47.345027 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 02:34:47.345060 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:34:47.345129 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:47.342634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38921 (local address 127.0.98.1:37939)
0928 02:34:47.342684 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:47.342686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:47.342765 (+ 79us) server_negotiation.cc:407] Connection header received
0928 02:34:47.342846 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:47.342847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:47.342861 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:47.342905 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:47.343077 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:47.343523 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:47.344476 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:47.344821 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:47.344875 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:47.344886 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:47.345032 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 02:34:47.345067 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 02:34:47.345139 4970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:47.342568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34901 (local address 127.0.98.2:49977)
0928 02:34:47.342733 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:47.342743 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:47.342803 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:47.342932 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:47.342933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:47.342935 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:47.342997 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:47.342999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:47.343606 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:47.343607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:47.344380 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:47.344382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:47.344895 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:47.344896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:47.344964 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:47.344968 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:34:47.345011 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 02:34:47.345050 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":13,"thread_start_us":69,"threads_started":1}
I20260928 02:34:47.345149 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:47.342567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37939 (local address 127.0.98.2:38921)
0928 02:34:47.342698 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:47.342709 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:47.342769 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:47.342909 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:47.342910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:47.342912 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:47.343011 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:47.343012 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:47.343591 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:47.343592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:47.344388 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:47.344390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:47.344888 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:47.344889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:47.344958 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:47.344962 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:34:47.345016 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 02:34:47.345057 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
I20260928 02:34:47.345500 4900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbdae08ee279472e840b058663752c17" candidate_uuid: "fb96c2270b934e9ab2990eae86476aa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee3a28f0c7c84c31957e05fc25e4b349" is_pre_election: true
I20260928 02:34:47.345548 4634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbdae08ee279472e840b058663752c17" candidate_uuid: "fb96c2270b934e9ab2990eae86476aa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcaa8422059f40e0bd40736a5db52dcf" is_pre_election: true
I20260928 02:34:47.345644 4900 raft_consensus.cc:2454] T bbdae08ee279472e840b058663752c17 P ee3a28f0c7c84c31957e05fc25e4b349 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb96c2270b934e9ab2990eae86476aa3 in term 0.
I20260928 02:34:47.345695 4634 raft_consensus.cc:2454] T bbdae08ee279472e840b058663752c17 P bcaa8422059f40e0bd40736a5db52dcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb96c2270b934e9ab2990eae86476aa3 in term 0.
I20260928 02:34:47.345896 4701 leader_election.cc:302] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcaa8422059f40e0bd40736a5db52dcf, fb96c2270b934e9ab2990eae86476aa3; no voters:
I20260928 02:34:47.346022 4968 raft_consensus.cc:2781] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:34:47.346076 4968 raft_consensus.cc:482] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:34:47.346097 4968 raft_consensus.cc:3037] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:47.346761 4968 raft_consensus.cc:504] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee3a28f0c7c84c31957e05fc25e4b349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34901 } } peers { permanent_uuid: "bcaa8422059f40e0bd40736a5db52dcf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37939 } } peers { permanent_uuid: "fb96c2270b934e9ab2990eae86476aa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } }
I20260928 02:34:47.346879 4968 leader_election.cc:288] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3 [CANDIDATE]: Term 1 election: Requested vote from peers ee3a28f0c7c84c31957e05fc25e4b349 (127.0.98.3:34901), bcaa8422059f40e0bd40736a5db52dcf (127.0.98.1:37939)
I20260928 02:34:47.347045 4900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbdae08ee279472e840b058663752c17" candidate_uuid: "fb96c2270b934e9ab2990eae86476aa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee3a28f0c7c84c31957e05fc25e4b349"
I20260928 02:34:47.347029 4634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbdae08ee279472e840b058663752c17" candidate_uuid: "fb96c2270b934e9ab2990eae86476aa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcaa8422059f40e0bd40736a5db52dcf"
I20260928 02:34:47.347113 4900 raft_consensus.cc:3037] T bbdae08ee279472e840b058663752c17 P ee3a28f0c7c84c31957e05fc25e4b349 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:47.347113 4634 raft_consensus.cc:3037] T bbdae08ee279472e840b058663752c17 P bcaa8422059f40e0bd40736a5db52dcf [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:47.347934 4634 raft_consensus.cc:2454] T bbdae08ee279472e840b058663752c17 P bcaa8422059f40e0bd40736a5db52dcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb96c2270b934e9ab2990eae86476aa3 in term 1.
I20260928 02:34:47.347934 4900 raft_consensus.cc:2454] T bbdae08ee279472e840b058663752c17 P ee3a28f0c7c84c31957e05fc25e4b349 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb96c2270b934e9ab2990eae86476aa3 in term 1.
I20260928 02:34:47.348091 4701 leader_election.cc:302] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcaa8422059f40e0bd40736a5db52dcf, fb96c2270b934e9ab2990eae86476aa3; no voters:
I20260928 02:34:47.348179 4968 raft_consensus.cc:2781] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:47.348304 4968 raft_consensus.cc:686] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3 [term 1 LEADER]: Becoming Leader. State: Replica: fb96c2270b934e9ab2990eae86476aa3, State: Running, Role: LEADER
I20260928 02:34:47.348382 4968 consensus_queue.cc:237] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee3a28f0c7c84c31957e05fc25e4b349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34901 } } peers { permanent_uuid: "bcaa8422059f40e0bd40736a5db52dcf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37939 } } peers { permanent_uuid: "fb96c2270b934e9ab2990eae86476aa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } }
I20260928 02:34:47.349123 4489 catalog_manager.cc:5495] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb96c2270b934e9ab2990eae86476aa3 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "fb96c2270b934e9ab2990eae86476aa3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee3a28f0c7c84c31957e05fc25e4b349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcaa8422059f40e0bd40736a5db52dcf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb96c2270b934e9ab2990eae86476aa3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43289 } health_report { overall_health: HEALTHY } } }
I20260928 02:34:47.429724 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:47.427498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52108 (local address 127.0.98.2:43289)
0928 02:34:47.427558 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:47.427560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:47.427568 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:34:47.427605 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:47.427606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:47.427626 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:47.427685 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:34:47.427834 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:47.428291 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:47.429023 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:47.429205 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:47.429283 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:47.429431 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:34:47.429536 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:34:47.429606 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:47.429629 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 02:34:47.429649 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 02:34:47.431381 4900 raft_consensus.cc:1261] T bbdae08ee279472e840b058663752c17 P ee3a28f0c7c84c31957e05fc25e4b349 [term 1 FOLLOWER]: Refusing update from remote peer fb96c2270b934e9ab2990eae86476aa3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:47.431401 4634 raft_consensus.cc:1261] T bbdae08ee279472e840b058663752c17 P bcaa8422059f40e0bd40736a5db52dcf [term 1 FOLLOWER]: Refusing update from remote peer fb96c2270b934e9ab2990eae86476aa3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:47.431730 4968 consensus_queue.cc:1035] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee3a28f0c7c84c31957e05fc25e4b349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:47.431859 4971 consensus_queue.cc:1035] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcaa8422059f40e0bd40736a5db52dcf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:47.433977 4974 mvcc.cc:205] Tried to move back new op lower bound from 7334145586916552704 to 7334145586579554304. Current Snapshot: MvccSnapshot[applied={T|T < 7334145586916552704}]
I20260928 02:34:47.435070 4975 mvcc.cc:205] Tried to move back new op lower bound from 7334145586916552704 to 7334145586579554304. Current Snapshot: MvccSnapshot[applied={T|T < 7334145586916552704 or (T in {7334145586916552704})}]
I20260928 02:34:47.435992 4489 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:34:47.436079 4489 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:34:47.436712 4489 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6328d0d4c309491e9c9b3e8c20ebe49a: Sending DeleteTablet for 3 replicas of tablet bbdae08ee279472e840b058663752c17
I20260928 02:34:47.436995 4747 tablet_service.cc:1514] Processing DeleteTablet for tablet bbdae08ee279472e840b058663752c17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52100
I20260928 02:34:47.437000 4614 tablet_service.cc:1514] Processing DeleteTablet for tablet bbdae08ee279472e840b058663752c17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60556
I20260928 02:34:47.437230 4983 tablet_replica.cc:330] T bbdae08ee279472e840b058663752c17 P bcaa8422059f40e0bd40736a5db52dcf: stopping tablet replica
I20260928 02:34:47.437202 4880 tablet_service.cc:1514] Processing DeleteTablet for tablet bbdae08ee279472e840b058663752c17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51430
I20260928 02:34:47.437379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 4552
I20260928 02:34:47.438076 4985 tablet_replica.cc:330] T bbdae08ee279472e840b058663752c17 P ee3a28f0c7c84c31957e05fc25e4b349: stopping tablet replica
I20260928 02:34:47.438212 4985 raft_consensus.cc:2229] T bbdae08ee279472e840b058663752c17 P ee3a28f0c7c84c31957e05fc25e4b349 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:34:47.438365 4985 raft_consensus.cc:2258] T bbdae08ee279472e840b058663752c17 P ee3a28f0c7c84c31957e05fc25e4b349 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:47.438585 4984 tablet_replica.cc:330] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3: stopping tablet replica
I20260928 02:34:47.438669 4984 raft_consensus.cc:2229] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:34:47.438752 4984 raft_consensus.cc:2258] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:47.439025 4985 ts_tablet_manager.cc:1905] T bbdae08ee279472e840b058663752c17 P ee3a28f0c7c84c31957e05fc25e4b349: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:47.439090 4984 ts_tablet_manager.cc:1905] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:47.440753 4985 ts_tablet_manager.cc:1918] T bbdae08ee279472e840b058663752c17 P ee3a28f0c7c84c31957e05fc25e4b349: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:34:47.440778 4984 ts_tablet_manager.cc:1918] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:34:47.440829 4984 log.cc:1196] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-1/wal/wals/bbdae08ee279472e840b058663752c17
I20260928 02:34:47.440909 4985 log.cc:1196] T bbdae08ee279472e840b058663752c17 P ee3a28f0c7c84c31957e05fc25e4b349: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790562862804661-392-0/minicluster-data/ts-2/wal/wals/bbdae08ee279472e840b058663752c17
I20260928 02:34:47.441092 4984 ts_tablet_manager.cc:1939] T bbdae08ee279472e840b058663752c17 P fb96c2270b934e9ab2990eae86476aa3: Deleting consensus metadata
I20260928 02:34:47.441214 4985 ts_tablet_manager.cc:1939] T bbdae08ee279472e840b058663752c17 P ee3a28f0c7c84c31957e05fc25e4b349: Deleting consensus metadata
I20260928 02:34:47.441511 4474 catalog_manager.cc:4841] TS fb96c2270b934e9ab2990eae86476aa3 (127.0.98.2:43289): tablet bbdae08ee279472e840b058663752c17 (table test-table [id=a9c6a422afa74067b6008d40e04eb511]) successfully deleted
I20260928 02:34:47.441577 4477 catalog_manager.cc:4841] TS ee3a28f0c7c84c31957e05fc25e4b349 (127.0.98.3:34901): tablet bbdae08ee279472e840b058663752c17 (table test-table [id=a9c6a422afa74067b6008d40e04eb511]) successfully deleted
W20260928 02:34:47.442636 4475 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37939 (error 108)
I20260928 02:34:47.442836 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 4685
I20260928 02:34:47.443002 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:47.442903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37939 (local address 127.0.0.1:60566)
0928 02:34:47.442945 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:47.442974 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 02:34:47.443629 4475 catalog_manager.cc:4584] TS bcaa8422059f40e0bd40736a5db52dcf (127.0.98.1:37939): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bbdae08ee279472e840b058663752c17: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37939: connect: Connection refused (error 111)
I20260928 02:34:47.447330 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 4818
I20260928 02:34:47.451192 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 4459
2026-09-28T02:34:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2140 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T02:34:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:34:47Z Disabled control of system clock
I20260928 02:34:47.475410 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39953
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 02:34:47.551278 4990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:47.551477 4990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:47.551496 4990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:47.552585 4990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:34:47.552624 4990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:47.552635 4990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:47.552647 4990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:34:47.552670 4990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:34:47.553987 4990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39953
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:47.554203 4990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:47.554456 4990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:47.558785 4998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:47.558871 4990 server_base.cc:1034] running on GCE node
W20260928 02:34:47.558977 4996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:47.559126 4995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:47.559333 4990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:47.559589 4990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:47.560725 4990 hybrid_clock.cc:648] HybridClock initialized: now 1790562887560717 us; error 25 us; skew 500 ppm
I20260928 02:34:47.561878 4990 webserver.cc:460] Webserver started at http://127.0.98.62:42403/ using document root <none> and password file <none>
I20260928 02:34:47.562098 4990 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:47.562141 4990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:47.562261 4990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:47.563257 4990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "9074c86a172341e3a18907bf0a0532d3"
format_stamp: "Formatted at 2026-09-28 02:34:47 on dist-test-slave-fz47"
server_key: "356ea230bb0f54bee1cf4c467453786a"
server_key_iv: "96e6e2693a585800b9f4d8ae7ac68616"
server_key_version: "encryptionkey@0"
I20260928 02:34:47.563568 4990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "9074c86a172341e3a18907bf0a0532d3"
format_stamp: "Formatted at 2026-09-28 02:34:47 on dist-test-slave-fz47"
server_key: "356ea230bb0f54bee1cf4c467453786a"
server_key_iv: "96e6e2693a585800b9f4d8ae7ac68616"
server_key_version: "encryptionkey@0"
I20260928 02:34:47.564674 4990 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:34:47.565296 5004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:47.565438 4990 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:47.565508 4990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "9074c86a172341e3a18907bf0a0532d3"
format_stamp: "Formatted at 2026-09-28 02:34:47 on dist-test-slave-fz47"
server_key: "356ea230bb0f54bee1cf4c467453786a"
server_key_iv: "96e6e2693a585800b9f4d8ae7ac68616"
server_key_version: "encryptionkey@0"
I20260928 02:34:47.565572 4990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:47.568590 4990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:47.568809 4990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:47.568918 4990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:47.572332 4990 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39953
I20260928 02:34:47.572371 5056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39953 every 8 connection(s)
I20260928 02:34:47.572643 4990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:34:47.573220 5057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:47.575762 5057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9074c86a172341e3a18907bf0a0532d3: Bootstrap starting.
I20260928 02:34:47.576257 5057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9074c86a172341e3a18907bf0a0532d3: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:47.576501 5057 log.cc:824] T 00000000000000000000000000000000 P 9074c86a172341e3a18907bf0a0532d3: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:47.577135 5057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9074c86a172341e3a18907bf0a0532d3: No bootstrap required, opened a new log
I20260928 02:34:47.578692 5057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9074c86a172341e3a18907bf0a0532d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9074c86a172341e3a18907bf0a0532d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39953 } }
I20260928 02:34:47.578819 5057 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9074c86a172341e3a18907bf0a0532d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:47.578815 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 4990
I20260928 02:34:47.578838 5057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9074c86a172341e3a18907bf0a0532d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9074c86a172341e3a18907bf0a0532d3, State: Initialized, Role: FOLLOWER
I20260928 02:34:47.578904 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:34:47.578963 5057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9074c86a172341e3a18907bf0a0532d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9074c86a172341e3a18907bf0a0532d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39953 } }
I20260928 02:34:47.579039 5057 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9074c86a172341e3a18907bf0a0532d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:34:47.579070 5057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9074c86a172341e3a18907bf0a0532d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:34:47.579102 392 external_mini_cluster.cc:1428] Setting key 1f44881a91257e94cbe5666c5e795240
I20260928 02:34:47.579115 5057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9074c86a172341e3a18907bf0a0532d3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:47.579730 5057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9074c86a172341e3a18907bf0a0532d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9074c86a172341e3a18907bf0a0532d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39953 } }
I20260928 02:34:47.579830 5057 leader_election.cc:302] T 00000000000000000000000000000000 P 9074c86a172341e3a18907bf0a0532d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9074c86a172341e3a18907bf0a0532d3; no voters:
I20260928 02:34:47.579981 5057 leader_election.cc:288] T 00000000000000000000000000000000 P 9074c86a172341e3a18907bf0a0532d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:34:47.580020 5062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9074c86a172341e3a18907bf0a0532d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:47.580169 5062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9074c86a172341e3a18907bf0a0532d3 [term 1 LEADER]: Becoming Leader. State: Replica: 9074c86a172341e3a18907bf0a0532d3, State: Running, Role: LEADER
I20260928 02:34:47.580201 5057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9074c86a172341e3a18907bf0a0532d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:34:47.580998 5062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9074c86a172341e3a18907bf0a0532d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9074c86a172341e3a18907bf0a0532d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39953 } }
I20260928 02:34:47.581377 5061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:47.579580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48782 (local address 127.0.98.62:39953)
0928 02:34:47.579809 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:47.579817 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:47.579861 (+ 44us) server_negotiation.cc:407] Connection header received
0928 02:34:47.579978 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:47.579985 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:47.580030 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:47.580168 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:47.580817 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:47.580820 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:47.580820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:34:47.580823 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:47.580874 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:34:47.580885 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:47.580933 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:47.581092 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 02:34:47.581169 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":102,"threads_started":1}
I20260928 02:34:47.581668 5064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9074c86a172341e3a18907bf0a0532d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9074c86a172341e3a18907bf0a0532d3. Latest consensus state: current_term: 1 leader_uuid: "9074c86a172341e3a18907bf0a0532d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9074c86a172341e3a18907bf0a0532d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39953 } } }
I20260928 02:34:47.581749 5064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9074c86a172341e3a18907bf0a0532d3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:47.581984 5063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9074c86a172341e3a18907bf0a0532d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9074c86a172341e3a18907bf0a0532d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9074c86a172341e3a18907bf0a0532d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39953 } } }
I20260928 02:34:47.582188 5063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9074c86a172341e3a18907bf0a0532d3 [sys.catalog]: This master's current role is: LEADER
W20260928 02:34:47.582471 5077 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9074c86a172341e3a18907bf0a0532d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:34:47.582520 5077 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:34:47.582556 5075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:34:47.582707 5075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:34:47.583984 5075 catalog_manager.cc:1347] Generated new cluster ID: 3b155e3cbc4547b2928ecdfaa11911de
I20260928 02:34:47.584028 5075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:34:47.605309 5075 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:34:47.605505 5075 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:34:47.607941 5075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9074c86a172341e3a18907bf0a0532d3: Generated new TSK 0
I20260928 02:34:47.608093 5075 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:34:47.611192 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39953
--builtin_ntp_servers=127.0.98.20:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 02:34:47.688933 5081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:47.689141 5081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:47.689170 5081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:47.690326 5081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:47.690366 5081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:47.690398 5081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:34:47.691699 5081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:47.691915 5081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:47.692174 5081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:47.697682 5086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:47.697688 5087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:47.697912 5081 server_base.cc:1034] running on GCE node
W20260928 02:34:47.697935 5089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:47.698118 5081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:47.698324 5081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:47.699465 5081 hybrid_clock.cc:648] HybridClock initialized: now 1790562887699454 us; error 31 us; skew 500 ppm
I20260928 02:34:47.700577 5081 webserver.cc:460] Webserver started at http://127.0.98.1:36333/ using document root <none> and password file <none>
I20260928 02:34:47.700778 5081 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:47.700825 5081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:47.700948 5081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:47.701987 5081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-0/data/instance:
uuid: "9132f9d16f724b83ac831b8ebe32d9ae"
format_stamp: "Formatted at 2026-09-28 02:34:47 on dist-test-slave-fz47"
server_key: "9ccaffc94ccbc2bfa10d94fa19ab6476"
server_key_iv: "8956b8273240627d980f5565e44ddfd8"
server_key_version: "encryptionkey@0"
I20260928 02:34:47.702296 5081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9132f9d16f724b83ac831b8ebe32d9ae"
format_stamp: "Formatted at 2026-09-28 02:34:47 on dist-test-slave-fz47"
server_key: "9ccaffc94ccbc2bfa10d94fa19ab6476"
server_key_iv: "8956b8273240627d980f5565e44ddfd8"
server_key_version: "encryptionkey@0"
I20260928 02:34:47.703421 5081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 02:34:47.704000 5095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:47.704162 5081 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:34:47.704236 5081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "9132f9d16f724b83ac831b8ebe32d9ae"
format_stamp: "Formatted at 2026-09-28 02:34:47 on dist-test-slave-fz47"
server_key: "9ccaffc94ccbc2bfa10d94fa19ab6476"
server_key_iv: "8956b8273240627d980f5565e44ddfd8"
server_key_version: "encryptionkey@0"
I20260928 02:34:47.704293 5081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:47.707513 5081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:47.707741 5081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:47.707852 5081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:47.708022 5081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:34:47.708282 5081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:47.708312 5081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:47.708336 5081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:47.708359 5081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:47.713469 5081 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37351
I20260928 02:34:47.713526 5208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37351 every 8 connection(s)
I20260928 02:34:47.713766 5081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
I20260928 02:34:47.715467 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 5081
I20260928 02:34:47.715556 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-0/wal/instance
I20260928 02:34:47.715709 392 external_mini_cluster.cc:1428] Setting key b6e0d5e366e1e8958b27bed033814e5c
I20260928 02:34:47.716432 5061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:47.714333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44199 (local address 127.0.98.62:39953)
0928 02:34:47.714381 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:47.714383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:47.714591 (+ 208us) server_negotiation.cc:407] Connection header received
0928 02:34:47.715016 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:47.715018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:47.715025 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:47.715074 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:47.716177 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:47.716178 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:47.716179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:34:47.716182 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:47.716199 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:34:47.716203 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:47.716254 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:47.716351 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 02:34:47.716365 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 02:34:47.716552 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39953
--builtin_ntp_servers=127.0.98.20:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 02:34:47.716575 5211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:47.714322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.1:44199)
0928 02:34:47.714490 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:47.714519 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:47.714930 (+ 411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:47.715136 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:47.715140 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:47.715150 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:47.716067 (+ 917us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:34:47.716068 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:47.716093 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:34:47.716119 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:47.716261 (+ 142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:47.716265 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:47.716267 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:34:47.716357 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 02:34:47.716402 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":69,"threads_started":1}
I20260928 02:34:47.716956 5209 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39953
I20260928 02:34:47.717034 5209 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:47.717160 5209 heartbeater.cc:511] Master 127.0.98.62:39953 requested a full tablet report, sending...
I20260928 02:34:47.717651 5021 ts_manager.cc:194] Registered new tserver with Master: 9132f9d16f724b83ac831b8ebe32d9ae (127.0.98.1:37351)
W20260928 02:34:47.792688 5212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:47.792865 5212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:47.792884 5212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:47.794018 5212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:47.794054 5212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:47.794076 5212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:34:47.795336 5212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:47.795531 5212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:47.795782 5212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:34:47.800284 5212 server_base.cc:1034] running on GCE node
W20260928 02:34:47.800222 5218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:47.800467 5217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:47.800616 5220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:47.800796 5212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:47.801013 5212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:47.802142 5212 hybrid_clock.cc:648] HybridClock initialized: now 1790562887802129 us; error 28 us; skew 500 ppm
I20260928 02:34:47.803282 5212 webserver.cc:460] Webserver started at http://127.0.98.2:40481/ using document root <none> and password file <none>
I20260928 02:34:47.803484 5212 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:47.803527 5212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:47.803640 5212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:47.804606 5212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-1/data/instance:
uuid: "2b83602b66484e46b1ca976be76f5526"
format_stamp: "Formatted at 2026-09-28 02:34:47 on dist-test-slave-fz47"
server_key: "ddf6a0eec4cf2a32fe27c06a138576ff"
server_key_iv: "2596f27aaab65a146f385d766ac2ede4"
server_key_version: "encryptionkey@0"
I20260928 02:34:47.804904 5212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2b83602b66484e46b1ca976be76f5526"
format_stamp: "Formatted at 2026-09-28 02:34:47 on dist-test-slave-fz47"
server_key: "ddf6a0eec4cf2a32fe27c06a138576ff"
server_key_iv: "2596f27aaab65a146f385d766ac2ede4"
server_key_version: "encryptionkey@0"
I20260928 02:34:47.806095 5212 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:34:47.806733 5226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:47.806871 5212 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:34:47.806943 5212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "2b83602b66484e46b1ca976be76f5526"
format_stamp: "Formatted at 2026-09-28 02:34:47 on dist-test-slave-fz47"
server_key: "ddf6a0eec4cf2a32fe27c06a138576ff"
server_key_iv: "2596f27aaab65a146f385d766ac2ede4"
server_key_version: "encryptionkey@0"
I20260928 02:34:47.807005 5212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:47.810150 5212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:47.810362 5212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:47.810477 5212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:47.810649 5212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:34:47.810918 5212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:47.810952 5212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:47.810983 5212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:47.811007 5212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:47.815941 5212 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46053
I20260928 02:34:47.815999 5339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46053 every 8 connection(s)
I20260928 02:34:47.816237 5212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
I20260928 02:34:47.818830 5061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:47.816816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55761 (local address 127.0.98.62:39953)
0928 02:34:47.816866 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:47.816868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:47.817072 (+ 204us) server_negotiation.cc:407] Connection header received
0928 02:34:47.817563 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:47.817565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:47.817571 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:47.817621 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:47.818610 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:47.818611 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:47.818611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:34:47.818616 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:47.818632 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:34:47.818636 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:47.818688 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:47.818750 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 02:34:47.818763 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 02:34:47.818998 5342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:47.816805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.2:55761)
0928 02:34:47.816971 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:47.817001 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:47.817475 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:47.817683 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:47.817687 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:47.817698 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:47.818482 (+ 784us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:34:47.818483 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:47.818507 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:34:47.818532 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:47.818688 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:47.818693 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:47.818696 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:34:47.818760 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 02:34:47.818814 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
I20260928 02:34:47.819375 5340 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39953
I20260928 02:34:47.819456 5340 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:47.819578 5340 heartbeater.cc:511] Master 127.0.98.62:39953 requested a full tablet report, sending...
I20260928 02:34:47.819854 5021 ts_manager.cc:194] Registered new tserver with Master: 2b83602b66484e46b1ca976be76f5526 (127.0.98.2:46053)
I20260928 02:34:47.820672 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 5212
I20260928 02:34:47.820747 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-1/wal/instance
I20260928 02:34:47.820887 392 external_mini_cluster.cc:1428] Setting key f7dc8ac4eee50018d40dea4039af5cd5
I20260928 02:34:47.821668 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39953
--builtin_ntp_servers=127.0.98.20:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 02:34:47.896773 5343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:47.896960 5343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:47.896986 5343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:47.898100 5343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:47.898136 5343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:47.898167 5343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:34:47.899437 5343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:47.899655 5343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:47.899897 5343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:47.904410 5351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:47.904565 5349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:47.904517 5343 server_base.cc:1034] running on GCE node
W20260928 02:34:47.904630 5348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:47.904923 5343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:47.905112 5343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:47.906247 5343 hybrid_clock.cc:648] HybridClock initialized: now 1790562887906232 us; error 29 us; skew 500 ppm
I20260928 02:34:47.907382 5343 webserver.cc:460] Webserver started at http://127.0.98.3:44139/ using document root <none> and password file <none>
I20260928 02:34:47.907639 5343 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:47.907686 5343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:47.907778 5343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:47.908756 5343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-2/data/instance:
uuid: "d2e5d35c75614a8292d65648bf88ebd6"
format_stamp: "Formatted at 2026-09-28 02:34:47 on dist-test-slave-fz47"
server_key: "be08f1b7e31f3c0623ae5efd5d61a9d2"
server_key_iv: "8e41cf4fd57bb697fbd5e5f5f16bbfbd"
server_key_version: "encryptionkey@0"
I20260928 02:34:47.909036 5343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d2e5d35c75614a8292d65648bf88ebd6"
format_stamp: "Formatted at 2026-09-28 02:34:47 on dist-test-slave-fz47"
server_key: "be08f1b7e31f3c0623ae5efd5d61a9d2"
server_key_iv: "8e41cf4fd57bb697fbd5e5f5f16bbfbd"
server_key_version: "encryptionkey@0"
I20260928 02:34:47.910180 5343 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:34:47.910801 5357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:47.910938 5343 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:47.911006 5343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "d2e5d35c75614a8292d65648bf88ebd6"
format_stamp: "Formatted at 2026-09-28 02:34:47 on dist-test-slave-fz47"
server_key: "be08f1b7e31f3c0623ae5efd5d61a9d2"
server_key_iv: "8e41cf4fd57bb697fbd5e5f5f16bbfbd"
server_key_version: "encryptionkey@0"
I20260928 02:34:47.911055 5343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:47.914106 5343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:47.914297 5343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:47.914382 5343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:47.914516 5343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:34:47.914729 5343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:47.914757 5343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:47.914774 5343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:47.914790 5343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:47.919674 5343 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41925
I20260928 02:34:47.919731 5470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41925 every 8 connection(s)
I20260928 02:34:47.919968 5343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
I20260928 02:34:47.922389 5061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:47.920524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56149 (local address 127.0.98.62:39953)
0928 02:34:47.920574 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:47.920576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:47.920769 (+ 193us) server_negotiation.cc:407] Connection header received
0928 02:34:47.921158 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:47.921159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:47.921165 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:47.921214 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:47.922207 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:47.922208 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:47.922209 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:34:47.922212 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:47.922227 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:34:47.922230 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:47.922273 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:47.922326 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 02:34:47.922338 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 02:34:47.922546 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:47.920521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39953 (local address 127.0.98.3:56149)
0928 02:34:47.920663 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:47.920691 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:47.921085 (+ 394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:47.921294 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:47.921298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:47.921308 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:47.922085 (+ 777us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:34:47.922086 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:47.922111 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:34:47.922135 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:47.922276 (+ 141us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:47.922278 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:47.922280 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:34:47.922328 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 02:34:47.922371 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":70,"threads_started":1}
I20260928 02:34:47.922966 5471 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39953
I20260928 02:34:47.923043 5471 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:47.923171 5471 heartbeater.cc:511] Master 127.0.98.62:39953 requested a full tablet report, sending...
I20260928 02:34:47.923472 5021 ts_manager.cc:194] Registered new tserver with Master: d2e5d35c75614a8292d65648bf88ebd6 (127.0.98.3:41925)
I20260928 02:34:47.925220 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 5343
I20260928 02:34:47.925330 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-2/wal/instance
I20260928 02:34:47.925472 392 external_mini_cluster.cc:1428] Setting key 9422db9dc935162c098474d7774b83f8
I20260928 02:34:47.926103 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:34:47.927999 5061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:47.927218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48784 (local address 127.0.98.62:39953)
0928 02:34:47.927249 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:47.927251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:47.927323 (+ 72us) server_negotiation.cc:407] Connection header received
0928 02:34:47.927369 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:47.927370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:47.927373 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:47.927432 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:47.927853 (+ 421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:47.927854 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:47.927854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:34:47.927857 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:47.927869 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:34:47.927872 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:47.927916 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:47.927948 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 02:34:47.927958 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260928 02:34:47.929194 5021 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:34:47.930650 5021 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:34:47.934104 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:47.932467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46053 (local address 127.0.0.1:51982)
0928 02:34:47.932733 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:47.932742 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:47.932779 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:47.933439 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:47.933440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:47.933443 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:47.933719 (+ 276us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:34:47.933720 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:47.933727 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:34:47.933739 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:47.933972 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:47.933973 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:47.933974 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 02:34:47.934019 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 02:34:47.934031 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":73,"threads_started":1}
I20260928 02:34:47.934119 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:47.932538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51982 (local address 127.0.98.2:46053)
0928 02:34:47.933324 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:47.933328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:47.933337 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:34:47.933358 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:47.933360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:47.933368 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:47.933436 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:47.933814 (+ 378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:47.933823 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:47.933823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:34:47.933828 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:47.933861 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:34:47.933884 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:47.933972 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:47.934024 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 02:34:47.934038 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":81,"threads_started":1}
I20260928 02:34:47.934165 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:47.932804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54860 (local address 127.0.98.1:37351)
0928 02:34:47.932962 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:47.932966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:47.932973 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:34:47.932993 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:47.932995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:47.933002 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:47.933059 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:47.933814 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:47.933823 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:47.933823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:34:47.933828 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:47.933862 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:34:47.933884 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:47.933942 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:47.934070 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 02:34:47.934086 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
I20260928 02:34:47.934211 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:47.932453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41925 (local address 127.0.0.1:52316)
0928 02:34:47.932600 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:47.932634 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:47.932683 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:47.933088 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:47.933089 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:47.933091 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:47.933893 (+ 802us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:34:47.933893 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:47.933899 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:34:47.933909 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:47.934113 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:47.934113 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:47.934114 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 02:34:47.934150 (+ 36us) client_negotiation.cc:240] Negotiation successful
0928 02:34:47.934160 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":52,"threads_started":1}
I20260928 02:34:47.934412 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:47.932528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52316 (local address 127.0.98.3:41925)
0928 02:34:47.932966 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:47.932970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:47.932978 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:34:47.932997 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:47.932999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:47.933007 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:47.933071 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:47.933990 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:47.933995 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:47.933995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:34:47.933999 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:47.934027 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:34:47.934045 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:47.934092 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:47.934331 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 02:34:47.934345 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":65,"threads_started":1}
I20260928 02:34:47.934571 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:47.932628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37351 (local address 127.0.0.1:54860)
0928 02:34:47.932818 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:47.932826 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:47.932861 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:47.933076 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:47.933082 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:47.933087 (+ 5us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:47.933597 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:34:47.933598 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:47.933613 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:34:47.933633 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:47.933935 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:47.933941 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:47.933943 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:34:47.934001 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 02:34:47.934012 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":66,"threads_started":1}
I20260928 02:34:47.934924 5274 tablet_service.cc:1467] Processing CreateTablet for tablet 20c3461459ae4fcfb4b956837472452d (DEFAULT_TABLE table=test-table [id=7cfcd81d19e147859c7d59f544efe47a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:47.935000 5405 tablet_service.cc:1467] Processing CreateTablet for tablet 20c3461459ae4fcfb4b956837472452d (DEFAULT_TABLE table=test-table [id=7cfcd81d19e147859c7d59f544efe47a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:47.935134 5143 tablet_service.cc:1467] Processing CreateTablet for tablet 20c3461459ae4fcfb4b956837472452d (DEFAULT_TABLE table=test-table [id=7cfcd81d19e147859c7d59f544efe47a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:47.935254 5274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20c3461459ae4fcfb4b956837472452d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:47.935262 5405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20c3461459ae4fcfb4b956837472452d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:47.935424 5143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20c3461459ae4fcfb4b956837472452d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:47.937500 5485 tablet_bootstrap.cc:492] T 20c3461459ae4fcfb4b956837472452d P d2e5d35c75614a8292d65648bf88ebd6: Bootstrap starting.
I20260928 02:34:47.938194 5485 tablet_bootstrap.cc:654] T 20c3461459ae4fcfb4b956837472452d P d2e5d35c75614a8292d65648bf88ebd6: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:47.938227 5486 tablet_bootstrap.cc:492] T 20c3461459ae4fcfb4b956837472452d P 9132f9d16f724b83ac831b8ebe32d9ae: Bootstrap starting.
I20260928 02:34:47.938437 5487 tablet_bootstrap.cc:492] T 20c3461459ae4fcfb4b956837472452d P 2b83602b66484e46b1ca976be76f5526: Bootstrap starting.
I20260928 02:34:47.938567 5485 log.cc:824] T 20c3461459ae4fcfb4b956837472452d P d2e5d35c75614a8292d65648bf88ebd6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:47.939003 5487 tablet_bootstrap.cc:654] T 20c3461459ae4fcfb4b956837472452d P 2b83602b66484e46b1ca976be76f5526: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:47.939247 5485 tablet_bootstrap.cc:492] T 20c3461459ae4fcfb4b956837472452d P d2e5d35c75614a8292d65648bf88ebd6: No bootstrap required, opened a new log
I20260928 02:34:47.939291 5487 log.cc:824] T 20c3461459ae4fcfb4b956837472452d P 2b83602b66484e46b1ca976be76f5526: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:47.939311 5485 ts_tablet_manager.cc:1397] T 20c3461459ae4fcfb4b956837472452d P d2e5d35c75614a8292d65648bf88ebd6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:34:47.939875 5487 tablet_bootstrap.cc:492] T 20c3461459ae4fcfb4b956837472452d P 2b83602b66484e46b1ca976be76f5526: No bootstrap required, opened a new log
I20260928 02:34:47.939941 5487 ts_tablet_manager.cc:1397] T 20c3461459ae4fcfb4b956837472452d P 2b83602b66484e46b1ca976be76f5526: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:34:47.940362 5486 tablet_bootstrap.cc:654] T 20c3461459ae4fcfb4b956837472452d P 9132f9d16f724b83ac831b8ebe32d9ae: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:47.940697 5486 log.cc:824] T 20c3461459ae4fcfb4b956837472452d P 9132f9d16f724b83ac831b8ebe32d9ae: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:47.941501 5486 tablet_bootstrap.cc:492] T 20c3461459ae4fcfb4b956837472452d P 9132f9d16f724b83ac831b8ebe32d9ae: No bootstrap required, opened a new log
I20260928 02:34:47.941599 5486 ts_tablet_manager.cc:1397] T 20c3461459ae4fcfb4b956837472452d P 9132f9d16f724b83ac831b8ebe32d9ae: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:34:47.941747 5485 raft_consensus.cc:348] T 20c3461459ae4fcfb4b956837472452d P d2e5d35c75614a8292d65648bf88ebd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b83602b66484e46b1ca976be76f5526" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "d2e5d35c75614a8292d65648bf88ebd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41925 } } peers { permanent_uuid: "9132f9d16f724b83ac831b8ebe32d9ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37351 } }
I20260928 02:34:47.941905 5485 raft_consensus.cc:374] T 20c3461459ae4fcfb4b956837472452d P d2e5d35c75614a8292d65648bf88ebd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:47.941977 5485 raft_consensus.cc:729] T 20c3461459ae4fcfb4b956837472452d P d2e5d35c75614a8292d65648bf88ebd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2e5d35c75614a8292d65648bf88ebd6, State: Initialized, Role: FOLLOWER
I20260928 02:34:47.941890 5487 raft_consensus.cc:348] T 20c3461459ae4fcfb4b956837472452d P 2b83602b66484e46b1ca976be76f5526 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b83602b66484e46b1ca976be76f5526" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "d2e5d35c75614a8292d65648bf88ebd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41925 } } peers { permanent_uuid: "9132f9d16f724b83ac831b8ebe32d9ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37351 } }
I20260928 02:34:47.942066 5487 raft_consensus.cc:374] T 20c3461459ae4fcfb4b956837472452d P 2b83602b66484e46b1ca976be76f5526 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:47.942108 5487 raft_consensus.cc:729] T 20c3461459ae4fcfb4b956837472452d P 2b83602b66484e46b1ca976be76f5526 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b83602b66484e46b1ca976be76f5526, State: Initialized, Role: FOLLOWER
I20260928 02:34:47.942113 5485 consensus_queue.cc:260] T 20c3461459ae4fcfb4b956837472452d P d2e5d35c75614a8292d65648bf88ebd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b83602b66484e46b1ca976be76f5526" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "d2e5d35c75614a8292d65648bf88ebd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41925 } } peers { permanent_uuid: "9132f9d16f724b83ac831b8ebe32d9ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37351 } }
I20260928 02:34:47.942216 5487 consensus_queue.cc:260] T 20c3461459ae4fcfb4b956837472452d P 2b83602b66484e46b1ca976be76f5526 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b83602b66484e46b1ca976be76f5526" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "d2e5d35c75614a8292d65648bf88ebd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41925 } } peers { permanent_uuid: "9132f9d16f724b83ac831b8ebe32d9ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37351 } }
I20260928 02:34:47.942384 5485 ts_tablet_manager.cc:1428] T 20c3461459ae4fcfb4b956837472452d P d2e5d35c75614a8292d65648bf88ebd6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:34:47.942447 5487 ts_tablet_manager.cc:1428] T 20c3461459ae4fcfb4b956837472452d P 2b83602b66484e46b1ca976be76f5526: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 02:34:47.942466 5340 heartbeater.cc:503] Master 127.0.98.62:39953 was elected leader, sending a full tablet report...
I20260928 02:34:47.942515 5471 heartbeater.cc:503] Master 127.0.98.62:39953 was elected leader, sending a full tablet report...
I20260928 02:34:47.943904 5486 raft_consensus.cc:348] T 20c3461459ae4fcfb4b956837472452d P 9132f9d16f724b83ac831b8ebe32d9ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b83602b66484e46b1ca976be76f5526" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "d2e5d35c75614a8292d65648bf88ebd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41925 } } peers { permanent_uuid: "9132f9d16f724b83ac831b8ebe32d9ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37351 } }
I20260928 02:34:47.944017 5486 raft_consensus.cc:374] T 20c3461459ae4fcfb4b956837472452d P 9132f9d16f724b83ac831b8ebe32d9ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:47.944037 5486 raft_consensus.cc:729] T 20c3461459ae4fcfb4b956837472452d P 9132f9d16f724b83ac831b8ebe32d9ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9132f9d16f724b83ac831b8ebe32d9ae, State: Initialized, Role: FOLLOWER
I20260928 02:34:47.944131 5486 consensus_queue.cc:260] T 20c3461459ae4fcfb4b956837472452d P 9132f9d16f724b83ac831b8ebe32d9ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b83602b66484e46b1ca976be76f5526" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "d2e5d35c75614a8292d65648bf88ebd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41925 } } peers { permanent_uuid: "9132f9d16f724b83ac831b8ebe32d9ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37351 } }
I20260928 02:34:47.944330 5486 ts_tablet_manager.cc:1428] T 20c3461459ae4fcfb4b956837472452d P 9132f9d16f724b83ac831b8ebe32d9ae: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:34:47.944372 5209 heartbeater.cc:503] Master 127.0.98.62:39953 was elected leader, sending a full tablet report...
W20260928 02:34:47.964211 5210 tablet.cc:2369] T 20c3461459ae4fcfb4b956837472452d P 9132f9d16f724b83ac831b8ebe32d9ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:34:48.016630 5492 raft_consensus.cc:482] T 20c3461459ae4fcfb4b956837472452d P 2b83602b66484e46b1ca976be76f5526 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:34:48.016743 5492 raft_consensus.cc:504] T 20c3461459ae4fcfb4b956837472452d P 2b83602b66484e46b1ca976be76f5526 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b83602b66484e46b1ca976be76f5526" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "d2e5d35c75614a8292d65648bf88ebd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41925 } } peers { permanent_uuid: "9132f9d16f724b83ac831b8ebe32d9ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37351 } }
I20260928 02:34:48.017021 5492 leader_election.cc:288] T 20c3461459ae4fcfb4b956837472452d P 2b83602b66484e46b1ca976be76f5526 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2e5d35c75614a8292d65648bf88ebd6 (127.0.98.3:41925), 9132f9d16f724b83ac831b8ebe32d9ae (127.0.98.1:37351)
I20260928 02:34:48.018170 5342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:48.017181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41925 (local address 127.0.98.2:36123)
0928 02:34:48.017233 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:48.017243 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:48.017308 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:48.017491 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:48.017491 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:48.017494 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:48.017875 (+ 381us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:34:48.017876 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:48.017882 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:34:48.017907 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:48.018046 (+ 139us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:48.018047 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:48.018047 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 02:34:48.018092 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 02:34:48.018104 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 02:34:48.018194 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:48.017236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36123 (local address 127.0.98.3:41925)
0928 02:34:48.017289 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:48.017291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:48.017299 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:34:48.017369 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:48.017370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:48.017375 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:48.017434 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:48.017966 (+ 532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:48.017967 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:48.017968 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:34:48.017972 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:48.017988 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:34:48.017992 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:48.018041 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:48.018113 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 02:34:48.018125 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 02:34:48.018486 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:48.017317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37351 (local address 127.0.98.2:44997)
0928 02:34:48.017490 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:48.017497 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:48.017544 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:48.017674 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:48.017674 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:48.017676 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:48.018086 (+ 410us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:34:48.018087 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:48.018093 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:34:48.018102 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:48.018345 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:48.018345 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:48.018346 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 02:34:48.018410 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 02:34:48.018431 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":118,"threads_started":1}
I20260928 02:34:48.018498 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:48.017386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44997 (local address 127.0.98.1:37351)
0928 02:34:48.017457 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:48.017459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:48.017552 (+ 93us) server_negotiation.cc:407] Connection header received
0928 02:34:48.017597 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:48.017598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:48.017604 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:48.017656 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:48.018173 (+ 517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:48.018174 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:48.018175 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:34:48.018179 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:48.018195 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:34:48.018199 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:48.018255 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:48.018424 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 02:34:48.018440 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:34:48.018487 5425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c3461459ae4fcfb4b956837472452d" candidate_uuid: "2b83602b66484e46b1ca976be76f5526" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2e5d35c75614a8292d65648bf88ebd6" is_pre_election: true
I20260928 02:34:48.018628 5425 raft_consensus.cc:2454] T 20c3461459ae4fcfb4b956837472452d P d2e5d35c75614a8292d65648bf88ebd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b83602b66484e46b1ca976be76f5526 in term 0.
I20260928 02:34:48.018708 5163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c3461459ae4fcfb4b956837472452d" candidate_uuid: "2b83602b66484e46b1ca976be76f5526" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9132f9d16f724b83ac831b8ebe32d9ae" is_pre_election: true
I20260928 02:34:48.018831 5163 raft_consensus.cc:2454] T 20c3461459ae4fcfb4b956837472452d P 9132f9d16f724b83ac831b8ebe32d9ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b83602b66484e46b1ca976be76f5526 in term 0.
I20260928 02:34:48.018841 5229 leader_election.cc:302] T 20c3461459ae4fcfb4b956837472452d P 2b83602b66484e46b1ca976be76f5526 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b83602b66484e46b1ca976be76f5526, d2e5d35c75614a8292d65648bf88ebd6; no voters:
I20260928 02:34:48.018965 5492 raft_consensus.cc:2781] T 20c3461459ae4fcfb4b956837472452d P 2b83602b66484e46b1ca976be76f5526 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:34:48.019032 5492 raft_consensus.cc:482] T 20c3461459ae4fcfb4b956837472452d P 2b83602b66484e46b1ca976be76f5526 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:34:48.019062 5492 raft_consensus.cc:3037] T 20c3461459ae4fcfb4b956837472452d P 2b83602b66484e46b1ca976be76f5526 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:48.019721 5492 raft_consensus.cc:504] T 20c3461459ae4fcfb4b956837472452d P 2b83602b66484e46b1ca976be76f5526 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b83602b66484e46b1ca976be76f5526" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "d2e5d35c75614a8292d65648bf88ebd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41925 } } peers { permanent_uuid: "9132f9d16f724b83ac831b8ebe32d9ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37351 } }
I20260928 02:34:48.019824 5492 leader_election.cc:288] T 20c3461459ae4fcfb4b956837472452d P 2b83602b66484e46b1ca976be76f5526 [CANDIDATE]: Term 1 election: Requested vote from peers d2e5d35c75614a8292d65648bf88ebd6 (127.0.98.3:41925), 9132f9d16f724b83ac831b8ebe32d9ae (127.0.98.1:37351)
I20260928 02:34:48.019977 5425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c3461459ae4fcfb4b956837472452d" candidate_uuid: "2b83602b66484e46b1ca976be76f5526" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2e5d35c75614a8292d65648bf88ebd6"
I20260928 02:34:48.019990 5163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c3461459ae4fcfb4b956837472452d" candidate_uuid: "2b83602b66484e46b1ca976be76f5526" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9132f9d16f724b83ac831b8ebe32d9ae"
I20260928 02:34:48.020051 5425 raft_consensus.cc:3037] T 20c3461459ae4fcfb4b956837472452d P d2e5d35c75614a8292d65648bf88ebd6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:48.020051 5163 raft_consensus.cc:3037] T 20c3461459ae4fcfb4b956837472452d P 9132f9d16f724b83ac831b8ebe32d9ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:48.020855 5163 raft_consensus.cc:2454] T 20c3461459ae4fcfb4b956837472452d P 9132f9d16f724b83ac831b8ebe32d9ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b83602b66484e46b1ca976be76f5526 in term 1.
I20260928 02:34:48.020855 5425 raft_consensus.cc:2454] T 20c3461459ae4fcfb4b956837472452d P d2e5d35c75614a8292d65648bf88ebd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b83602b66484e46b1ca976be76f5526 in term 1.
I20260928 02:34:48.021000 5229 leader_election.cc:302] T 20c3461459ae4fcfb4b956837472452d P 2b83602b66484e46b1ca976be76f5526 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b83602b66484e46b1ca976be76f5526, d2e5d35c75614a8292d65648bf88ebd6; no voters:
I20260928 02:34:48.021098 5492 raft_consensus.cc:2781] T 20c3461459ae4fcfb4b956837472452d P 2b83602b66484e46b1ca976be76f5526 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:48.021214 5492 raft_consensus.cc:686] T 20c3461459ae4fcfb4b956837472452d P 2b83602b66484e46b1ca976be76f5526 [term 1 LEADER]: Becoming Leader. State: Replica: 2b83602b66484e46b1ca976be76f5526, State: Running, Role: LEADER
I20260928 02:34:48.021306 5492 consensus_queue.cc:237] T 20c3461459ae4fcfb4b956837472452d P 2b83602b66484e46b1ca976be76f5526 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b83602b66484e46b1ca976be76f5526" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "d2e5d35c75614a8292d65648bf88ebd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41925 } } peers { permanent_uuid: "9132f9d16f724b83ac831b8ebe32d9ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37351 } }
I20260928 02:34:48.022087 5020 catalog_manager.cc:5495] T 20c3461459ae4fcfb4b956837472452d P 2b83602b66484e46b1ca976be76f5526 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b83602b66484e46b1ca976be76f5526 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2b83602b66484e46b1ca976be76f5526" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b83602b66484e46b1ca976be76f5526" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2e5d35c75614a8292d65648bf88ebd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9132f9d16f724b83ac831b8ebe32d9ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37351 } health_report { overall_health: UNKNOWN } } }
I20260928 02:34:48.026787 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:48.026109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51994 (local address 127.0.98.2:46053)
0928 02:34:48.026154 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:48.026156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:48.026165 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:34:48.026222 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:48.026223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:48.026231 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:48.026274 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:48.026658 (+ 384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:48.026658 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:48.026659 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:34:48.026662 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:48.026673 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:34:48.026676 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:48.026714 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:48.026740 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 02:34:48.026749 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 02:34:48.028162 5163 raft_consensus.cc:1261] T 20c3461459ae4fcfb4b956837472452d P 9132f9d16f724b83ac831b8ebe32d9ae [term 1 FOLLOWER]: Refusing update from remote peer 2b83602b66484e46b1ca976be76f5526: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:48.028189 5425 raft_consensus.cc:1261] T 20c3461459ae4fcfb4b956837472452d P d2e5d35c75614a8292d65648bf88ebd6 [term 1 FOLLOWER]: Refusing update from remote peer 2b83602b66484e46b1ca976be76f5526: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:48.028416 5495 consensus_queue.cc:1035] T 20c3461459ae4fcfb4b956837472452d P 2b83602b66484e46b1ca976be76f5526 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9132f9d16f724b83ac831b8ebe32d9ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:48.028519 5492 consensus_queue.cc:1035] T 20c3461459ae4fcfb4b956837472452d P 2b83602b66484e46b1ca976be76f5526 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2e5d35c75614a8292d65648bf88ebd6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:48.030584 5498 mvcc.cc:205] Tried to move back new op lower bound from 7334145589361553408 to 7334145589335932928. Current Snapshot: MvccSnapshot[applied={T|T < 7334145589361553408}]
I20260928 02:34:48.031503 5500 mvcc.cc:205] Tried to move back new op lower bound from 7334145589361553408 to 7334145589335932928. Current Snapshot: MvccSnapshot[applied={T|T < 7334145589361553408 or (T in {7334145589361553408})}]
I20260928 02:34:48.031639 5499 mvcc.cc:205] Tried to move back new op lower bound from 7334145589361553408 to 7334145589335932928. Current Snapshot: MvccSnapshot[applied={T|T < 7334145589361553408 or (T in {7334145589361553408})}]
I20260928 02:34:48.032693 5019 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:34:48.032768 5019 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:34:48.033420 5019 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9074c86a172341e3a18907bf0a0532d3: Sending DeleteTablet for 3 replicas of tablet 20c3461459ae4fcfb4b956837472452d
I20260928 02:34:48.033710 5274 tablet_service.cc:1514] Processing DeleteTablet for tablet 20c3461459ae4fcfb4b956837472452d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:48 UTC) from {username='slave'} at 127.0.0.1:51982
I20260928 02:34:48.033751 5405 tablet_service.cc:1514] Processing DeleteTablet for tablet 20c3461459ae4fcfb4b956837472452d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:48 UTC) from {username='slave'} at 127.0.0.1:52316
I20260928 02:34:48.033771 5143 tablet_service.cc:1514] Processing DeleteTablet for tablet 20c3461459ae4fcfb4b956837472452d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:48 UTC) from {username='slave'} at 127.0.0.1:54860
I20260928 02:34:48.034011 5509 tablet_replica.cc:330] T 20c3461459ae4fcfb4b956837472452d P 9132f9d16f724b83ac831b8ebe32d9ae: stopping tablet replica
I20260928 02:34:48.034065 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 5081
I20260928 02:34:48.034122 5509 raft_consensus.cc:2229] T 20c3461459ae4fcfb4b956837472452d P 9132f9d16f724b83ac831b8ebe32d9ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:34:48.034224 5509 raft_consensus.cc:2258] T 20c3461459ae4fcfb4b956837472452d P 9132f9d16f724b83ac831b8ebe32d9ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:48.037372 5507 tablet_replica.cc:330] T 20c3461459ae4fcfb4b956837472452d P 2b83602b66484e46b1ca976be76f5526: stopping tablet replica
I20260928 02:34:48.037475 5507 raft_consensus.cc:2229] T 20c3461459ae4fcfb4b956837472452d P 2b83602b66484e46b1ca976be76f5526 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:34:48.037559 5507 raft_consensus.cc:2258] T 20c3461459ae4fcfb4b956837472452d P 2b83602b66484e46b1ca976be76f5526 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:48.037693 5508 tablet_replica.cc:330] T 20c3461459ae4fcfb4b956837472452d P d2e5d35c75614a8292d65648bf88ebd6: stopping tablet replica
I20260928 02:34:48.037822 5508 raft_consensus.cc:2229] T 20c3461459ae4fcfb4b956837472452d P d2e5d35c75614a8292d65648bf88ebd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:34:48.037945 5508 raft_consensus.cc:2258] T 20c3461459ae4fcfb4b956837472452d P d2e5d35c75614a8292d65648bf88ebd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:48.037992 5507 ts_tablet_manager.cc:1905] T 20c3461459ae4fcfb4b956837472452d P 2b83602b66484e46b1ca976be76f5526: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:48.038369 5508 ts_tablet_manager.cc:1905] T 20c3461459ae4fcfb4b956837472452d P d2e5d35c75614a8292d65648bf88ebd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:48.039808 5508 ts_tablet_manager.cc:1918] T 20c3461459ae4fcfb4b956837472452d P d2e5d35c75614a8292d65648bf88ebd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260928 02:34:48.039794 5007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37351 (error 108)
I20260928 02:34:48.039933 5508 log.cc:1196] T 20c3461459ae4fcfb4b956837472452d P d2e5d35c75614a8292d65648bf88ebd6: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790562862804661-392-0/minicluster-data/ts-2/wal/wals/20c3461459ae4fcfb4b956837472452d
I20260928 02:34:48.039965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 5212
I20260928 02:34:48.040688 5508 ts_tablet_manager.cc:1939] T 20c3461459ae4fcfb4b956837472452d P d2e5d35c75614a8292d65648bf88ebd6: Deleting consensus metadata
I20260928 02:34:48.041882 5007 catalog_manager.cc:4841] TS d2e5d35c75614a8292d65648bf88ebd6 (127.0.98.3:41925): tablet 20c3461459ae4fcfb4b956837472452d (table test-table [id=7cfcd81d19e147859c7d59f544efe47a]) successfully deleted
I20260928 02:34:48.041961 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:48.041835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37351 (local address 127.0.0.1:54866)
0928 02:34:48.041886 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:48.041930 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 02:34:48.042068 5007 catalog_manager.cc:4584] TS 9132f9d16f724b83ac831b8ebe32d9ae (127.0.98.1:37351): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 20c3461459ae4fcfb4b956837472452d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37351: connect: Connection refused (error 111)
I20260928 02:34:48.044494 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:48.044410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46053 (local address 127.0.0.1:52002)
0928 02:34:48.044450 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:48.044471 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 02:34:48.044588 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 5343
I20260928 02:34:48.048060 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 4990
2026-09-28T02:34:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (596 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 02:34:48.065606 392 mini_oidc.cc:152] Starting JWKS server
I20260928 02:34:48.066570 392 webserver.cc:460] Webserver started at https://127.0.0.1:35489/ using document root <none> and password file <none>
I20260928 02:34:48.066644 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:34:48.066759 392 webserver.cc:460] Webserver started at http://127.0.0.1:44003/ using document root <none> and password file <none>
2026-09-28T02:34:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:34:48Z Disabled control of system clock
I20260928 02:34:48.073555 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41255
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41255
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35489/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790562862804661-392-0/testchainca.cert with env {}
W20260928 02:34:48.149739 5516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:48.149927 5516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:48.149947 5516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:48.151024 5516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:34:48.151057 5516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:48.151068 5516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:48.151079 5516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:34:48.151094 5516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:34:48.151105 5516 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:34:48.151121 5516 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35489/jwks.json
I20260928 02:34:48.152550 5516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41255
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35489/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790562862804661-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:48.152724 5516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:48.152876 5516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:48.157150 5522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:48.157315 5524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:48.157408 5521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:48.157450 5516 server_base.cc:1034] running on GCE node
I20260928 02:34:48.157678 5516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:48.157963 5516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:48.159104 5516 hybrid_clock.cc:648] HybridClock initialized: now 1790562888159091 us; error 30 us; skew 500 ppm
I20260928 02:34:48.160220 5516 webserver.cc:460] Webserver started at http://127.0.98.62:36157/ using document root <none> and password file <none>
I20260928 02:34:48.160425 5516 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:48.160467 5516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:48.160578 5516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:48.161572 5516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "e8b5e3f99d9d4a0899206d2c2828ba80"
format_stamp: "Formatted at 2026-09-28 02:34:48 on dist-test-slave-fz47"
server_key: "2255ad3e0a8d1fc1babe80c4ad72fb61"
server_key_iv: "191abcf29f8db8d16944a0003d546718"
server_key_version: "encryptionkey@0"
I20260928 02:34:48.161880 5516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "e8b5e3f99d9d4a0899206d2c2828ba80"
format_stamp: "Formatted at 2026-09-28 02:34:48 on dist-test-slave-fz47"
server_key: "2255ad3e0a8d1fc1babe80c4ad72fb61"
server_key_iv: "191abcf29f8db8d16944a0003d546718"
server_key_version: "encryptionkey@0"
I20260928 02:34:48.162972 5516 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:34:48.163547 5530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:48.163671 5516 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:48.163748 5516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "e8b5e3f99d9d4a0899206d2c2828ba80"
format_stamp: "Formatted at 2026-09-28 02:34:48 on dist-test-slave-fz47"
server_key: "2255ad3e0a8d1fc1babe80c4ad72fb61"
server_key_iv: "191abcf29f8db8d16944a0003d546718"
server_key_version: "encryptionkey@0"
I20260928 02:34:48.163802 5516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:48.178565 5516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:48.178822 5516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:48.178942 5516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:48.182420 5516 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41255
I20260928 02:34:48.182449 5582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41255 every 8 connection(s)
I20260928 02:34:48.182752 5516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:34:48.183619 5583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:48.185739 5583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8b5e3f99d9d4a0899206d2c2828ba80: Bootstrap starting.
I20260928 02:34:48.186246 5583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8b5e3f99d9d4a0899206d2c2828ba80: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:48.186473 5583 log.cc:824] T 00000000000000000000000000000000 P e8b5e3f99d9d4a0899206d2c2828ba80: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:48.187023 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 5516
I20260928 02:34:48.187117 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:34:48.187134 5583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8b5e3f99d9d4a0899206d2c2828ba80: No bootstrap required, opened a new log
I20260928 02:34:48.187280 392 external_mini_cluster.cc:1428] Setting key 087f871420a735eb9094aaee8758d14b
I20260928 02:34:48.189669 5583 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8b5e3f99d9d4a0899206d2c2828ba80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b5e3f99d9d4a0899206d2c2828ba80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41255 } }
I20260928 02:34:48.189823 5583 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8b5e3f99d9d4a0899206d2c2828ba80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:48.189858 5583 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8b5e3f99d9d4a0899206d2c2828ba80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8b5e3f99d9d4a0899206d2c2828ba80, State: Initialized, Role: FOLLOWER
I20260928 02:34:48.189981 5583 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8b5e3f99d9d4a0899206d2c2828ba80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b5e3f99d9d4a0899206d2c2828ba80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41255 } }
I20260928 02:34:48.190052 5583 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8b5e3f99d9d4a0899206d2c2828ba80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:34:48.190085 5583 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8b5e3f99d9d4a0899206d2c2828ba80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:34:48.190128 5583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8b5e3f99d9d4a0899206d2c2828ba80 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:48.190771 5583 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8b5e3f99d9d4a0899206d2c2828ba80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b5e3f99d9d4a0899206d2c2828ba80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41255 } }
I20260928 02:34:48.190876 5583 leader_election.cc:302] T 00000000000000000000000000000000 P e8b5e3f99d9d4a0899206d2c2828ba80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8b5e3f99d9d4a0899206d2c2828ba80; no voters:
I20260928 02:34:48.191030 5583 leader_election.cc:288] T 00000000000000000000000000000000 P e8b5e3f99d9d4a0899206d2c2828ba80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:34:48.191051 5588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8b5e3f99d9d4a0899206d2c2828ba80 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:48.191146 5588 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8b5e3f99d9d4a0899206d2c2828ba80 [term 1 LEADER]: Becoming Leader. State: Replica: e8b5e3f99d9d4a0899206d2c2828ba80, State: Running, Role: LEADER
I20260928 02:34:48.191311 5583 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8b5e3f99d9d4a0899206d2c2828ba80 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:34:48.191295 5588 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8b5e3f99d9d4a0899206d2c2828ba80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b5e3f99d9d4a0899206d2c2828ba80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41255 } }
I20260928 02:34:48.191733 5590 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8b5e3f99d9d4a0899206d2c2828ba80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8b5e3f99d9d4a0899206d2c2828ba80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b5e3f99d9d4a0899206d2c2828ba80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41255 } } }
I20260928 02:34:48.191953 5590 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8b5e3f99d9d4a0899206d2c2828ba80 [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:48.191888 5589 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8b5e3f99d9d4a0899206d2c2828ba80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8b5e3f99d9d4a0899206d2c2828ba80. Latest consensus state: current_term: 1 leader_uuid: "e8b5e3f99d9d4a0899206d2c2828ba80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8b5e3f99d9d4a0899206d2c2828ba80" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41255 } } }
I20260928 02:34:48.192215 5589 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8b5e3f99d9d4a0899206d2c2828ba80 [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:48.192426 5597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:34:48.192842 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:48.187727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56390 (local address 127.0.98.62:41255)
0928 02:34:48.187894 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:48.187901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:48.187925 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:34:48.188423 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:48.188430 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:48.188487 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:48.188595 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:48.188833 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:48.189733 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:48.190791 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:48.191386 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:48.191535 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:48.192198 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:48.192202 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:48.192202 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:34:48.192395 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:48.192437 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:34:48.192450 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:48.192504 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:48.192575 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 02:34:48.192648 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260928 02:34:48.192950 5597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:34:48.193709 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:34:48.195086 5597 catalog_manager.cc:1347] Generated new cluster ID: 80af416cf30545aeb5c547f3aeafb20c
I20260928 02:34:48.195135 5597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:34:48.213228 5597 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:34:48.213721 5597 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:34:48.217235 5597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8b5e3f99d9d4a0899206d2c2828ba80: Generated new TSK 0
I20260928 02:34:48.217485 5597 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 02:34:48.244684 5613 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 02:34:48.244766 5613 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 02:34:48.244969 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:48.230468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56406 (local address 127.0.98.62:41255)
0928 02:34:48.230507 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:48.230509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:48.230608 (+ 99us) server_negotiation.cc:407] Connection header received
0928 02:34:48.230706 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:48.230706 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:48.230716 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:48.230767 (+ 51us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:34:48.230919 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:48.231401 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:48.232503 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:48.232678 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:48.232715 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:48.232918 (+ 203us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:34:48.244761 (+ 11843us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35489/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 02:34:48.244910 (+ 149us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56406: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35489/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 02:34:48.245021 5587 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56406: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35489/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 02:34:48.245343 5612 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:34:48.230377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41255 (local address 127.0.0.1:56406)
0928 02:34:48.230542 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:48.230553 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:48.230623 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:48.230793 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:48.230794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:48.230796 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:34:48.230865 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:48.230867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:48.231485 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:48.231486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:48.232432 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:48.232435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:48.232734 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:48.232740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:48.232808 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:48.232827 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:34:48.244922 (+ 12095us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35489/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 02:34:48.245012 (+ 90us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41255: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35489/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":75,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:41255: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35489/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 02:34:48.245720 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 5516
2026-09-28T02:34:48Z chronyd exiting
I20260928 02:34:48.280742 392 test_util.cc:175] -----------------------------------------------
I20260928 02:34:48.280812 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790562862804661-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (216 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 02:34:48.281524 392 mini_oidc.cc:152] Starting JWKS server
I20260928 02:34:48.282135 392 webserver.cc:460] Webserver started at https://127.0.0.1:37231/ using document root <none> and password file <none>
I20260928 02:34:48.282176 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:34:48.282277 392 webserver.cc:460] Webserver started at http://127.0.0.1:36231/ using document root <none> and password file <none>
2026-09-28T02:34:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:34:48Z Disabled control of system clock
I20260928 02:34:48.288661 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38571
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38571
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37231/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790562862804661-392-0/testchainca.cert with env {}
W20260928 02:34:48.363369 5619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:48.363567 5619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:48.363596 5619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:48.364717 5619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:34:48.364755 5619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:48.364771 5619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:48.364786 5619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:34:48.364801 5619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:34:48.364817 5619 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:34:48.364835 5619 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37231/jwks.json
I20260928 02:34:48.366278 5619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38571
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37231/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790562862804661-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:48.366509 5619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:48.366685 5619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:48.370924 5627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:48.370947 5624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:48.370949 5625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:48.371124 5619 server_base.cc:1034] running on GCE node
I20260928 02:34:48.371335 5619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:48.371582 5619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:48.372705 5619 hybrid_clock.cc:648] HybridClock initialized: now 1790562888372696 us; error 24 us; skew 500 ppm
I20260928 02:34:48.373795 5619 webserver.cc:460] Webserver started at http://127.0.98.62:45425/ using document root <none> and password file <none>
I20260928 02:34:48.373983 5619 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:48.374024 5619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:48.374135 5619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:48.375061 5619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "0a0861677b304464854058e441da3f35"
format_stamp: "Formatted at 2026-09-28 02:34:48 on dist-test-slave-fz47"
server_key: "a349ee14c5b7ef8aa8a1c1e5e63a0062"
server_key_iv: "b3bfdd70fbac721959a4b3fb96a596f2"
server_key_version: "encryptionkey@0"
I20260928 02:34:48.375376 5619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "0a0861677b304464854058e441da3f35"
format_stamp: "Formatted at 2026-09-28 02:34:48 on dist-test-slave-fz47"
server_key: "a349ee14c5b7ef8aa8a1c1e5e63a0062"
server_key_iv: "b3bfdd70fbac721959a4b3fb96a596f2"
server_key_version: "encryptionkey@0"
I20260928 02:34:48.376493 5619 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:34:48.377081 5633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:48.377225 5619 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:48.377322 5619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "0a0861677b304464854058e441da3f35"
format_stamp: "Formatted at 2026-09-28 02:34:48 on dist-test-slave-fz47"
server_key: "a349ee14c5b7ef8aa8a1c1e5e63a0062"
server_key_iv: "b3bfdd70fbac721959a4b3fb96a596f2"
server_key_version: "encryptionkey@0"
I20260928 02:34:48.377380 5619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:48.390053 5619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:48.390292 5619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:48.390419 5619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:48.393906 5619 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38571
I20260928 02:34:48.393935 5685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38571 every 8 connection(s)
I20260928 02:34:48.394220 5619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:34:48.395068 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:48.397130 5686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a0861677b304464854058e441da3f35: Bootstrap starting.
I20260928 02:34:48.397673 5686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a0861677b304464854058e441da3f35: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:48.397893 5686 log.cc:824] T 00000000000000000000000000000000 P 0a0861677b304464854058e441da3f35: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:48.398447 5686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a0861677b304464854058e441da3f35: No bootstrap required, opened a new log
I20260928 02:34:48.399899 5686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a0861677b304464854058e441da3f35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0861677b304464854058e441da3f35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38571 } }
I20260928 02:34:48.400027 5686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a0861677b304464854058e441da3f35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:48.400048 5686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a0861677b304464854058e441da3f35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a0861677b304464854058e441da3f35, State: Initialized, Role: FOLLOWER
I20260928 02:34:48.400146 5686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a0861677b304464854058e441da3f35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0861677b304464854058e441da3f35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38571 } }
I20260928 02:34:48.400190 5686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a0861677b304464854058e441da3f35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:34:48.400213 5686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a0861677b304464854058e441da3f35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:34:48.400240 5686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a0861677b304464854058e441da3f35 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:48.400666 5686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a0861677b304464854058e441da3f35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0861677b304464854058e441da3f35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38571 } }
I20260928 02:34:48.400743 5686 leader_election.cc:302] T 00000000000000000000000000000000 P 0a0861677b304464854058e441da3f35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a0861677b304464854058e441da3f35; no voters:
I20260928 02:34:48.400852 5686 leader_election.cc:288] T 00000000000000000000000000000000 P 0a0861677b304464854058e441da3f35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:34:48.400923 5689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a0861677b304464854058e441da3f35 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:48.401063 5689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a0861677b304464854058e441da3f35 [term 1 LEADER]: Becoming Leader. State: Replica: 0a0861677b304464854058e441da3f35, State: Running, Role: LEADER
I20260928 02:34:48.401068 5686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a0861677b304464854058e441da3f35 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:34:48.401168 5689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a0861677b304464854058e441da3f35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0861677b304464854058e441da3f35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38571 } }
I20260928 02:34:48.401551 5690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a0861677b304464854058e441da3f35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a0861677b304464854058e441da3f35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0861677b304464854058e441da3f35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38571 } } }
I20260928 02:34:48.401654 5691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a0861677b304464854058e441da3f35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a0861677b304464854058e441da3f35. Latest consensus state: current_term: 1 leader_uuid: "0a0861677b304464854058e441da3f35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a0861677b304464854058e441da3f35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38571 } } }
I20260928 02:34:48.401731 5691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a0861677b304464854058e441da3f35 [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:48.401925 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 5619
I20260928 02:34:48.402014 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:34:48.401684 5690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a0861677b304464854058e441da3f35 [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:48.402153 5698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:34:48.402225 392 external_mini_cluster.cc:1428] Setting key 8963c43eef9dc5a0828bebcfcc102a48
I20260928 02:34:48.402645 5698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:34:48.404052 5698 catalog_manager.cc:1347] Generated new cluster ID: 60b5f52dd91b4633a26f716d495294f3
I20260928 02:34:48.404103 5698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:34:48.408430 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:48.402788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48534 (local address 127.0.98.62:38571)
0928 02:34:48.403031 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:48.403039 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:48.403265 (+ 226us) server_negotiation.cc:407] Connection header received
0928 02:34:48.403392 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:48.403398 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:48.403464 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:48.403582 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:48.403913 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:48.405224 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:48.406279 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:48.406910 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:48.407026 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:48.407648 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:48.407652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:48.407653 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:34:48.407854 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:48.407896 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:34:48.407911 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:48.407976 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:48.408030 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 02:34:48.408201 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":62,"threads_started":1}
I20260928 02:34:48.409245 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:34:48.419085 5698 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:34:48.419760 5698 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:34:48.428673 5698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a0861677b304464854058e441da3f35: Generated new TSK 0
I20260928 02:34:48.428853 5698 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 02:34:48.453611 5716 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 02:34:48.453681 5716 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 02:34:48.453944 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:48.440148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48538 (local address 127.0.98.62:38571)
0928 02:34:48.440193 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:48.440196 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:48.440261 (+ 65us) server_negotiation.cc:407] Connection header received
0928 02:34:48.440333 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:48.440334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:48.440344 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:48.440402 (+ 58us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:34:48.440518 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:48.441015 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:48.442115 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:48.442307 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:48.442348 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:48.442510 (+ 162us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:34:48.453780 (+ 11270us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37231/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 02:34:48.453897 (+ 117us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48538: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37231/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 02:34:48.453984 5708 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48538: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37231/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 02:34:48.454088 5715 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:34:48.440064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38571 (local address 127.0.0.1:48538)
0928 02:34:48.440194 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:48.440204 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:48.440267 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:48.440408 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:48.440409 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:48.440411 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:34:48.440464 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:48.440466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:48.441077 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:48.441078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:48.442042 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:48.442044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:48.442358 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:48.442371 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:48.442431 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:48.442436 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:34:48.453877 (+ 11441us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37231/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 02:34:48.453992 (+ 115us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38571: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37231/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
I20260928 02:34:48.454236 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 5619
2026-09-28T02:34:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (209 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 02:34:48.490590 392 mini_oidc.cc:152] Starting JWKS server
I20260928 02:34:48.491199 392 webserver.cc:460] Webserver started at https://127.0.0.1:41865/ using document root <none> and password file <none>
I20260928 02:34:48.491250 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:34:48.491355 392 webserver.cc:460] Webserver started at http://127.0.0.1:46503/ using document root <none> and password file <none>
2026-09-28T02:34:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:34:48Z Disabled control of system clock
I20260928 02:34:48.497908 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40297
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40297
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41865/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 02:34:48.572721 5722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:48.572907 5722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:48.572926 5722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:48.574026 5722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:34:48.574059 5722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:48.574070 5722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:48.574081 5722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:34:48.574091 5722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:34:48.574106 5722 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:34:48.574116 5722 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41865/jwks.json
I20260928 02:34:48.575387 5722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40297
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41865/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:48.575564 5722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:48.575783 5722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:48.580085 5730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:48.580133 5722 server_base.cc:1034] running on GCE node
W20260928 02:34:48.580240 5728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:48.580282 5727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:48.580543 5722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:48.580816 5722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:48.581948 5722 hybrid_clock.cc:648] HybridClock initialized: now 1790562888581935 us; error 32 us; skew 500 ppm
I20260928 02:34:48.583021 5722 webserver.cc:460] Webserver started at http://127.0.98.62:34599/ using document root <none> and password file <none>
I20260928 02:34:48.583222 5722 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:48.583267 5722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:48.583393 5722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:48.584354 5722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "a94abdc9560646fc97099b6799b9e332"
format_stamp: "Formatted at 2026-09-28 02:34:48 on dist-test-slave-fz47"
server_key: "81c7c05e1be4de0924bda91c2776293a"
server_key_iv: "57a539fc2ee15a9fb7ff583c86b038e9"
server_key_version: "encryptionkey@0"
I20260928 02:34:48.584678 5722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "a94abdc9560646fc97099b6799b9e332"
format_stamp: "Formatted at 2026-09-28 02:34:48 on dist-test-slave-fz47"
server_key: "81c7c05e1be4de0924bda91c2776293a"
server_key_iv: "57a539fc2ee15a9fb7ff583c86b038e9"
server_key_version: "encryptionkey@0"
I20260928 02:34:48.585884 5722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:34:48.586503 5736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:48.586647 5722 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:48.586720 5722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "a94abdc9560646fc97099b6799b9e332"
format_stamp: "Formatted at 2026-09-28 02:34:48 on dist-test-slave-fz47"
server_key: "81c7c05e1be4de0924bda91c2776293a"
server_key_iv: "57a539fc2ee15a9fb7ff583c86b038e9"
server_key_version: "encryptionkey@0"
I20260928 02:34:48.586778 5722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:48.602876 5722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:48.603165 5722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:48.603292 5722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:48.606767 5722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40297
I20260928 02:34:48.606798 5788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40297 every 8 connection(s)
I20260928 02:34:48.607110 5722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:34:48.607733 5789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:48.610327 5789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a94abdc9560646fc97099b6799b9e332: Bootstrap starting.
I20260928 02:34:48.610858 5789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a94abdc9560646fc97099b6799b9e332: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:48.611107 5789 log.cc:824] T 00000000000000000000000000000000 P a94abdc9560646fc97099b6799b9e332: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:48.611291 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 5722
I20260928 02:34:48.611384 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:34:48.611533 392 external_mini_cluster.cc:1428] Setting key abedea7431cef4230e9783360d5c0310
I20260928 02:34:48.611743 5789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a94abdc9560646fc97099b6799b9e332: No bootstrap required, opened a new log
I20260928 02:34:48.614027 5789 raft_consensus.cc:348] T 00000000000000000000000000000000 P a94abdc9560646fc97099b6799b9e332 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a94abdc9560646fc97099b6799b9e332" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40297 } }
I20260928 02:34:48.614157 5789 raft_consensus.cc:374] T 00000000000000000000000000000000 P a94abdc9560646fc97099b6799b9e332 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:48.614179 5789 raft_consensus.cc:729] T 00000000000000000000000000000000 P a94abdc9560646fc97099b6799b9e332 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a94abdc9560646fc97099b6799b9e332, State: Initialized, Role: FOLLOWER
I20260928 02:34:48.614305 5789 consensus_queue.cc:260] T 00000000000000000000000000000000 P a94abdc9560646fc97099b6799b9e332 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a94abdc9560646fc97099b6799b9e332" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40297 } }
I20260928 02:34:48.614382 5789 raft_consensus.cc:388] T 00000000000000000000000000000000 P a94abdc9560646fc97099b6799b9e332 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:34:48.614410 5789 raft_consensus.cc:482] T 00000000000000000000000000000000 P a94abdc9560646fc97099b6799b9e332 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:34:48.614456 5789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a94abdc9560646fc97099b6799b9e332 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:48.614951 5789 raft_consensus.cc:504] T 00000000000000000000000000000000 P a94abdc9560646fc97099b6799b9e332 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a94abdc9560646fc97099b6799b9e332" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40297 } }
I20260928 02:34:48.615041 5789 leader_election.cc:302] T 00000000000000000000000000000000 P a94abdc9560646fc97099b6799b9e332 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a94abdc9560646fc97099b6799b9e332; no voters:
I20260928 02:34:48.615185 5789 leader_election.cc:288] T 00000000000000000000000000000000 P a94abdc9560646fc97099b6799b9e332 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:34:48.615226 5794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a94abdc9560646fc97099b6799b9e332 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:48.615329 5794 raft_consensus.cc:686] T 00000000000000000000000000000000 P a94abdc9560646fc97099b6799b9e332 [term 1 LEADER]: Becoming Leader. State: Replica: a94abdc9560646fc97099b6799b9e332, State: Running, Role: LEADER
I20260928 02:34:48.615406 5794 consensus_queue.cc:237] T 00000000000000000000000000000000 P a94abdc9560646fc97099b6799b9e332 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a94abdc9560646fc97099b6799b9e332" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40297 } }
I20260928 02:34:48.615433 5789 sys_catalog.cc:564] T 00000000000000000000000000000000 P a94abdc9560646fc97099b6799b9e332 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:34:48.615705 5794 sys_catalog.cc:455] T 00000000000000000000000000000000 P a94abdc9560646fc97099b6799b9e332 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a94abdc9560646fc97099b6799b9e332. Latest consensus state: current_term: 1 leader_uuid: "a94abdc9560646fc97099b6799b9e332" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a94abdc9560646fc97099b6799b9e332" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40297 } } }
I20260928 02:34:48.615768 5794 sys_catalog.cc:458] T 00000000000000000000000000000000 P a94abdc9560646fc97099b6799b9e332 [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:48.615914 5796 sys_catalog.cc:455] T 00000000000000000000000000000000 P a94abdc9560646fc97099b6799b9e332 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a94abdc9560646fc97099b6799b9e332" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a94abdc9560646fc97099b6799b9e332" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40297 } } }
I20260928 02:34:48.615970 5796 sys_catalog.cc:458] T 00000000000000000000000000000000 P a94abdc9560646fc97099b6799b9e332 [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:48.616073 5799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:34:48.616703 5799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:34:48.617398 5793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:48.611971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46048 (local address 127.0.98.62:40297)
0928 02:34:48.612180 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:48.612188 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:48.612213 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:34:48.612876 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:48.612892 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:48.612965 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:48.613077 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:48.613311 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:48.614173 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:48.614959 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:48.615505 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:48.615951 (+ 446us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:48.616356 (+ 405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:48.616359 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:48.616360 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:34:48.616562 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:48.616603 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:34:48.616618 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:48.616945 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:48.616975 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 02:34:48.617167 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":83,"threads_started":1}
I20260928 02:34:48.618201 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:34:48.618232 5799 catalog_manager.cc:1347] Generated new cluster ID: a322c244a8f14f10b6b7cb306b322365
I20260928 02:34:48.618280 5799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:34:48.634027 5799 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:34:48.634548 5799 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:34:48.637997 5799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a94abdc9560646fc97099b6799b9e332: Generated new TSK 0
I20260928 02:34:48.638139 5799 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:34:48.650097 5793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:48.646732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46064 (local address 127.0.98.62:40297)
0928 02:34:48.646785 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:48.646787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:48.646885 (+ 98us) server_negotiation.cc:407] Connection header received
0928 02:34:48.646967 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:48.646968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:48.646978 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:48.647033 (+ 55us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:34:48.647157 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:48.647674 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:48.648755 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:48.648948 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:48.648993 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:48.649148 (+ 155us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:34:48.649897 (+ 749us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41865/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 02:34:48.650012 (+ 115us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46064: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41865/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 02:34:48.650161 5793 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46064: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41865/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 02:34:48.650171 5818 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:34:48.646671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40297 (local address 127.0.0.1:46064)
0928 02:34:48.646818 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:48.646829 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:48.646904 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:48.647045 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:48.647045 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:48.647048 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:34:48.647103 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:48.647105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:48.647742 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:48.647743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:48.648687 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:48.648689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:48.649005 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:48.649012 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:48.649068 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:48.649074 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:34:48.649982 (+ 908us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41865/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 02:34:48.650066 (+ 84us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40297: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41865/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
I20260928 02:34:48.650575 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 5722
E20260928 02:34:48.661469 5819 webserver.cc:533] Webserver: sslize failed:
E20260928 02:34:48.661537 5819 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T02:34:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (196 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T02:34:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:34:48Z Disabled control of system clock
I20260928 02:34:48.693928 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34251
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34251
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790562862804661-392-0/testchainca.cert with env {}
W20260928 02:34:48.769459 5823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:48.769649 5823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:48.769676 5823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:48.770805 5823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:34:48.770843 5823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:48.770859 5823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:48.770874 5823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:34:48.770888 5823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:34:48.770905 5823 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:34:48.770922 5823 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 02:34:48.772370 5823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34251
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790562862804661-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:48.772596 5823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:48.772789 5823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:48.777127 5829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:48.777128 5828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:48.777149 5831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:48.777350 5823 server_base.cc:1034] running on GCE node
I20260928 02:34:48.777586 5823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:48.777830 5823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:48.778973 5823 hybrid_clock.cc:648] HybridClock initialized: now 1790562888778958 us; error 29 us; skew 500 ppm
I20260928 02:34:48.780051 5823 webserver.cc:460] Webserver started at http://127.0.98.62:37863/ using document root <none> and password file <none>
I20260928 02:34:48.780256 5823 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:48.780301 5823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:48.780416 5823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:48.781419 5823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "8ec4bb52869d4b68a31063132730a17a"
format_stamp: "Formatted at 2026-09-28 02:34:48 on dist-test-slave-fz47"
server_key: "a4ba125d5e67c91db6ca3fd3203c809c"
server_key_iv: "29596befc1340a51c1807972acad6dfc"
server_key_version: "encryptionkey@0"
I20260928 02:34:48.781720 5823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "8ec4bb52869d4b68a31063132730a17a"
format_stamp: "Formatted at 2026-09-28 02:34:48 on dist-test-slave-fz47"
server_key: "a4ba125d5e67c91db6ca3fd3203c809c"
server_key_iv: "29596befc1340a51c1807972acad6dfc"
server_key_version: "encryptionkey@0"
I20260928 02:34:48.782836 5823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:34:48.783489 5837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:48.783641 5823 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:48.783718 5823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "8ec4bb52869d4b68a31063132730a17a"
format_stamp: "Formatted at 2026-09-28 02:34:48 on dist-test-slave-fz47"
server_key: "a4ba125d5e67c91db6ca3fd3203c809c"
server_key_iv: "29596befc1340a51c1807972acad6dfc"
server_key_version: "encryptionkey@0"
I20260928 02:34:48.783774 5823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:48.790629 5823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:48.790894 5823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:48.791007 5823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:48.794454 5823 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34251
I20260928 02:34:48.794488 5889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34251 every 8 connection(s)
I20260928 02:34:48.794770 5823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:34:48.795622 5890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:48.797437 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 5823
I20260928 02:34:48.797545 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:34:48.797729 392 external_mini_cluster.cc:1428] Setting key 8e903877744de3379ce015f90a16aab6
I20260928 02:34:48.797953 5890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ec4bb52869d4b68a31063132730a17a: Bootstrap starting.
I20260928 02:34:48.798574 5890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ec4bb52869d4b68a31063132730a17a: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:48.798821 5890 log.cc:824] T 00000000000000000000000000000000 P 8ec4bb52869d4b68a31063132730a17a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:48.799407 5890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ec4bb52869d4b68a31063132730a17a: No bootstrap required, opened a new log
I20260928 02:34:48.801070 5890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ec4bb52869d4b68a31063132730a17a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec4bb52869d4b68a31063132730a17a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34251 } }
I20260928 02:34:48.801199 5890 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ec4bb52869d4b68a31063132730a17a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:48.801220 5890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ec4bb52869d4b68a31063132730a17a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ec4bb52869d4b68a31063132730a17a, State: Initialized, Role: FOLLOWER
I20260928 02:34:48.801349 5890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ec4bb52869d4b68a31063132730a17a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec4bb52869d4b68a31063132730a17a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34251 } }
I20260928 02:34:48.801438 5890 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ec4bb52869d4b68a31063132730a17a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:34:48.801478 5890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ec4bb52869d4b68a31063132730a17a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:34:48.801518 5890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ec4bb52869d4b68a31063132730a17a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:48.802076 5890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ec4bb52869d4b68a31063132730a17a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec4bb52869d4b68a31063132730a17a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34251 } }
I20260928 02:34:48.802161 5890 leader_election.cc:302] T 00000000000000000000000000000000 P 8ec4bb52869d4b68a31063132730a17a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ec4bb52869d4b68a31063132730a17a; no voters:
I20260928 02:34:48.802300 5890 leader_election.cc:288] T 00000000000000000000000000000000 P 8ec4bb52869d4b68a31063132730a17a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:34:48.802338 5895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ec4bb52869d4b68a31063132730a17a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:48.802475 5895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ec4bb52869d4b68a31063132730a17a [term 1 LEADER]: Becoming Leader. State: Replica: 8ec4bb52869d4b68a31063132730a17a, State: Running, Role: LEADER
I20260928 02:34:48.802549 5890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ec4bb52869d4b68a31063132730a17a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:34:48.802560 5895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ec4bb52869d4b68a31063132730a17a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec4bb52869d4b68a31063132730a17a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34251 } }
I20260928 02:34:48.802834 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:48.798170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40426 (local address 127.0.98.62:34251)
0928 02:34:48.798363 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:48.798370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:48.798393 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:34:48.798829 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:48.798836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:48.798910 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:48.799047 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:48.799232 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:48.800123 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:48.800850 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:48.801301 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:48.801420 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:48.802017 (+ 597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:48.802020 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:48.802020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:34:48.802185 (+ 165us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:48.802226 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:34:48.802240 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:48.802292 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:48.802573 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 02:34:48.802652 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20260928 02:34:48.803074 5897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ec4bb52869d4b68a31063132730a17a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ec4bb52869d4b68a31063132730a17a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec4bb52869d4b68a31063132730a17a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34251 } } }
I20260928 02:34:48.803218 5897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ec4bb52869d4b68a31063132730a17a [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:48.803236 5895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ec4bb52869d4b68a31063132730a17a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ec4bb52869d4b68a31063132730a17a. Latest consensus state: current_term: 1 leader_uuid: "8ec4bb52869d4b68a31063132730a17a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec4bb52869d4b68a31063132730a17a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34251 } } }
I20260928 02:34:48.803328 5895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ec4bb52869d4b68a31063132730a17a [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:48.803524 5904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:34:48.803655 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:34:48.804029 5904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:34:48.805524 5904 catalog_manager.cc:1347] Generated new cluster ID: 46cbb20eea1945318d3af4d0d5074076
I20260928 02:34:48.805575 5904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:34:48.815477 5904 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:34:48.816129 5904 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:34:48.823513 5904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ec4bb52869d4b68a31063132730a17a: Generated new TSK 0
I20260928 02:34:48.823673 5904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:34:48.907044 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:48.839334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40428 (local address 127.0.98.62:34251)
0928 02:34:48.839386 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:48.839388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:48.839443 (+ 55us) server_negotiation.cc:407] Connection header received
0928 02:34:48.839504 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:48.839504 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:48.839515 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:48.839593 (+ 78us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:34:48.839683 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:48.840467 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:48.841190 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:48.841525 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:48.841586 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:48.841728 (+ 142us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:34:48.906778 (+ 65050us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 02:34:48.906963 (+ 185us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40428: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":32}
W20260928 02:34:48.907130 5893 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40428: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 02:34:48.907141 5919 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:34:48.839259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34251 (local address 127.0.0.1:40428)
0928 02:34:48.839390 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:48.839400 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:48.839453 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:48.839591 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:48.839591 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:48.839593 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:34:48.839646 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:48.839647 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:48.840544 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:48.840544 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:48.841124 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:48.841125 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:48.841601 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:48.841603 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:48.841643 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:48.841648 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:34:48.906938 (+ 65290us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 02:34:48.907057 (+ 119us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34251: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
I20260928 02:34:48.907490 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 5823
2026-09-28T02:34:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (236 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (85 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (83 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:34:49 dist-test-slave-fz47 krb5kdc[5931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:34:49 dist-test-slave-fz47 krb5kdc[5931](info): set up 2 sockets
Sep 28 02:34:49 dist-test-slave-fz47 krb5kdc[5931](info): commencing operation
krb5kdc: starting...
W20260928 02:34:50.033105 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:34:50 dist-test-slave-fz47 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:34:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:34:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 02:34:50.141882 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44483
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44483
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:50.219274 5944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:50.219465 5944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:50.219484 5944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:50.220595 5944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:34:50.220634 5944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:50.220647 5944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:50.220669 5944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:34:50.220692 5944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:34:50.222170 5944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:50.222381 5944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:50.222573 5944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:50.226982 5949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:50.226984 5952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:50.227059 5950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:50.227222 5944 server_base.cc:1034] running on GCE node
I20260928 02:34:50.227466 5944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:50.227698 5944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:50.228833 5944 hybrid_clock.cc:648] HybridClock initialized: now 1790562890228820 us; error 27 us; skew 500 ppm
Sep 28 02:34:50 dist-test-slave-fz47 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:50.231065 5944 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:34:50.231470 5944 webserver.cc:460] Webserver started at http://127.0.98.62:38899/ using document root <none> and password file <none>
I20260928 02:34:50.231648 5944 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:50.231688 5944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:50.231812 5944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:50.232805 5944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "6f846ee56cb54d2e8f2dab3fd12a775f"
format_stamp: "Formatted at 2026-09-28 02:34:50 on dist-test-slave-fz47"
server_key: "e047d43b1122704739ba6bd522d48f31"
server_key_iv: "2377aae096357fb714d514c2f6ada0d1"
server_key_version: "encryptionkey@0"
I20260928 02:34:50.233121 5944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "6f846ee56cb54d2e8f2dab3fd12a775f"
format_stamp: "Formatted at 2026-09-28 02:34:50 on dist-test-slave-fz47"
server_key: "e047d43b1122704739ba6bd522d48f31"
server_key_iv: "2377aae096357fb714d514c2f6ada0d1"
server_key_version: "encryptionkey@0"
I20260928 02:34:50.234333 5944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:34:50.234962 5959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:50.235102 5944 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:34:50.235174 5944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "6f846ee56cb54d2e8f2dab3fd12a775f"
format_stamp: "Formatted at 2026-09-28 02:34:50 on dist-test-slave-fz47"
server_key: "e047d43b1122704739ba6bd522d48f31"
server_key_iv: "2377aae096357fb714d514c2f6ada0d1"
server_key_version: "encryptionkey@0"
I20260928 02:34:50.235236 5944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:50.255752 5944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:50.257525 5944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:50.257647 5944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:50.261127 5944 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44483
I20260928 02:34:50.261160 6011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44483 every 8 connection(s)
I20260928 02:34:50.261638 5944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:34:50.262305 6012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:50.264534 6012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f846ee56cb54d2e8f2dab3fd12a775f: Bootstrap starting.
I20260928 02:34:50.265105 6012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f846ee56cb54d2e8f2dab3fd12a775f: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:50.265409 6012 log.cc:824] T 00000000000000000000000000000000 P 6f846ee56cb54d2e8f2dab3fd12a775f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:50.265848 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 5944
I20260928 02:34:50.265961 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:34:50.266129 6012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f846ee56cb54d2e8f2dab3fd12a775f: No bootstrap required, opened a new log
I20260928 02:34:50.266211 392 external_mini_cluster.cc:1428] Setting key ca6dfe113b085a6d139041ff08fea51b
I20260928 02:34:50.268788 6012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f846ee56cb54d2e8f2dab3fd12a775f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f846ee56cb54d2e8f2dab3fd12a775f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44483 } }
I20260928 02:34:50.268929 6012 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f846ee56cb54d2e8f2dab3fd12a775f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:50.268951 6012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f846ee56cb54d2e8f2dab3fd12a775f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f846ee56cb54d2e8f2dab3fd12a775f, State: Initialized, Role: FOLLOWER
I20260928 02:34:50.269058 6012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f846ee56cb54d2e8f2dab3fd12a775f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f846ee56cb54d2e8f2dab3fd12a775f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44483 } }
I20260928 02:34:50.269127 6012 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f846ee56cb54d2e8f2dab3fd12a775f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:34:50.269150 6012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f846ee56cb54d2e8f2dab3fd12a775f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:34:50.269196 6012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f846ee56cb54d2e8f2dab3fd12a775f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:50.269899 6012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f846ee56cb54d2e8f2dab3fd12a775f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f846ee56cb54d2e8f2dab3fd12a775f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44483 } }
I20260928 02:34:50.270011 6012 leader_election.cc:302] T 00000000000000000000000000000000 P 6f846ee56cb54d2e8f2dab3fd12a775f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f846ee56cb54d2e8f2dab3fd12a775f; no voters:
I20260928 02:34:50.270170 6012 leader_election.cc:288] T 00000000000000000000000000000000 P 6f846ee56cb54d2e8f2dab3fd12a775f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:34:50.270200 6017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f846ee56cb54d2e8f2dab3fd12a775f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:50.270336 6017 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f846ee56cb54d2e8f2dab3fd12a775f [term 1 LEADER]: Becoming Leader. State: Replica: 6f846ee56cb54d2e8f2dab3fd12a775f, State: Running, Role: LEADER
I20260928 02:34:50.270411 6012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f846ee56cb54d2e8f2dab3fd12a775f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:34:50.270442 6017 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f846ee56cb54d2e8f2dab3fd12a775f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f846ee56cb54d2e8f2dab3fd12a775f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44483 } }
I20260928 02:34:50.270753 6017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f846ee56cb54d2e8f2dab3fd12a775f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f846ee56cb54d2e8f2dab3fd12a775f. Latest consensus state: current_term: 1 leader_uuid: "6f846ee56cb54d2e8f2dab3fd12a775f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f846ee56cb54d2e8f2dab3fd12a775f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44483 } } }
I20260928 02:34:50.270834 6017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f846ee56cb54d2e8f2dab3fd12a775f [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:50.270766 6019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f846ee56cb54d2e8f2dab3fd12a775f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f846ee56cb54d2e8f2dab3fd12a775f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f846ee56cb54d2e8f2dab3fd12a775f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44483 } } }
I20260928 02:34:50.271045 6019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f846ee56cb54d2e8f2dab3fd12a775f [sys.catalog]: This master's current role is: LEADER
W20260928 02:34:50.272543 6032 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6f846ee56cb54d2e8f2dab3fd12a775f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:34:50.272614 6032 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:34:50.272881 6023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:34:50.273046 6023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 02:34:50 dist-test-slave-fz47 krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:50.274606 6023 catalog_manager.cc:1347] Generated new cluster ID: 6980d71700544c7eb13e8ad2344fff04
I20260928 02:34:50.274652 6023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:34:50.277235 6016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:50.266727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60024 (local address 127.0.98.62:44483)
0928 02:34:50.266897 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:50.266908 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:50.266935 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:34:50.267428 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:50.267435 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:50.267481 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:50.267592 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:50.268330 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:50.269372 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:50.270363 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:50.270981 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:50.271091 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:50.273674 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:50.273679 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:50.273679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:50.273691 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:50.275522 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:50.275877 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:50.275880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:50.275881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:50.275969 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:50.276119 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:50.276120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:50.276120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:50.276397 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:50.276520 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:50.276905 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 02:34:50.277028 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":61,"threads_started":1}
I20260928 02:34:50.326056 6023 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:34:50.326762 6023 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:34:50.329856 6023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f846ee56cb54d2e8f2dab3fd12a775f: Generated new TSK 0
I20260928 02:34:50.330013 6023 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 02:34:50.356559 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44483
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:50.433228 6038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:50.433450 6038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:50.433480 6038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:50.434576 6038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:50.434612 6038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:50.434646 6038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:34:50.436048 6038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:50.436275 6038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:50.436463 6038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:50.441044 6043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:50.441200 6044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:50.441222 6038 server_base.cc:1034] running on GCE node
W20260928 02:34:50.441375 6046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:50.441577 6038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:50.441764 6038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:50.442893 6038 hybrid_clock.cc:648] HybridClock initialized: now 1790562890442882 us; error 27 us; skew 500 ppm
Sep 28 02:34:50 dist-test-slave-fz47 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:50.445206 6038 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 02:34:50.445600 6038 webserver.cc:460] Webserver started at http://127.0.98.1:39197/ using document root <none> and password file <none>
I20260928 02:34:50.445777 6038 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:50.445815 6038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:50.445899 6038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:50.446825 6038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-0/data/instance:
uuid: "2b32e8f810184838a05ce342d841fedd"
format_stamp: "Formatted at 2026-09-28 02:34:50 on dist-test-slave-fz47"
server_key: "b673c815ecc3b4d7b97de5b87bb656c8"
server_key_iv: "fe73d6cd13136d744a8a3618771b5964"
server_key_version: "encryptionkey@0"
I20260928 02:34:50.447104 6038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2b32e8f810184838a05ce342d841fedd"
format_stamp: "Formatted at 2026-09-28 02:34:50 on dist-test-slave-fz47"
server_key: "b673c815ecc3b4d7b97de5b87bb656c8"
server_key_iv: "fe73d6cd13136d744a8a3618771b5964"
server_key_version: "encryptionkey@0"
I20260928 02:34:50.448220 6038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:34:50.448853 6053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:50.448984 6038 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:50.449062 6038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "2b32e8f810184838a05ce342d841fedd"
format_stamp: "Formatted at 2026-09-28 02:34:50 on dist-test-slave-fz47"
server_key: "b673c815ecc3b4d7b97de5b87bb656c8"
server_key_iv: "fe73d6cd13136d744a8a3618771b5964"
server_key_version: "encryptionkey@0"
I20260928 02:34:50.449118 6038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:50.463876 6038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:50.465888 6038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:50.466022 6038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:50.466225 6038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:34:50.466506 6038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:50.466538 6038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:50.466566 6038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:50.466588 6038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:50.472018 6038 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40733
I20260928 02:34:50.472052 6166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40733 every 8 connection(s)
I20260928 02:34:50.472371 6038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 02:34:50 dist-test-slave-fz47 krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:50.480947 6016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:50.473008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38381 (local address 127.0.98.62:44483)
0928 02:34:50.473059 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:50.473061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:50.473226 (+ 165us) server_negotiation.cc:407] Connection header received
0928 02:34:50.474191 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:50.474192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:50.474205 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:50.474253 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:50.474845 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:50.475354 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:50.476296 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:50.476502 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:50.476556 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:50.478523 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:50.478524 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:50.478525 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:50.478532 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:50.479731 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:50.480084 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:50.480085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:50.480086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:50.480137 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:50.480358 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:50.480359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:50.480359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:50.480467 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:50.480521 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:50.480778 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 02:34:50.480860 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 02:34:50.481092 6169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:50.472982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44483 (local address 127.0.98.1:38381)
0928 02:34:50.473114 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:50.473149 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:50.474077 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:50.474331 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:50.474334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:50.474571 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:50.474752 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:50.474756 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:50.475431 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:50.475433 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:50.476204 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:50.476206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:50.476567 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:50.476568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:50.476685 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:50.477126 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:50.477133 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:50.478437 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:50.479810 (+ 1373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:50.479816 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:50.479818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:50.480022 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:50.480188 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:50.480189 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:50.480189 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:50.480303 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:50.480519 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:50.480525 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:50.480656 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 02:34:50.480740 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 02:34:50.480886 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":65,"threads_started":1}
I20260928 02:34:50.481150 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 6038
I20260928 02:34:50.481281 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-0/wal/instance
I20260928 02:34:50.481484 392 external_mini_cluster.cc:1428] Setting key 9c59e23fc6e99efd9357cf92519c7ce2
I20260928 02:34:50.481539 6167 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44483
I20260928 02:34:50.481630 6167 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:50.481810 6167 heartbeater.cc:511] Master 127.0.98.62:44483 requested a full tablet report, sending...
I20260928 02:34:50.482267 5976 ts_manager.cc:194] Registered new tserver with Master: 2b32e8f810184838a05ce342d841fedd (127.0.98.1:40733)
I20260928 02:34:50.482828 5976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38381
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 02:34:50.501395 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44483
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:50.577525 6172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:50.577705 6172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:50.577726 6172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:50.578794 6172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:50.578825 6172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:50.578850 6172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:34:50.580228 6172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:50.580394 6172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:50.580557 6172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:50.585019 6177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:50.585135 6178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:50.585026 6180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:50.585289 6172 server_base.cc:1034] running on GCE node
I20260928 02:34:50.585494 6172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:50.585695 6172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:50.586840 6172 hybrid_clock.cc:648] HybridClock initialized: now 1790562890586825 us; error 27 us; skew 500 ppm
Sep 28 02:34:50 dist-test-slave-fz47 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:50.589184 6172 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 02:34:50.589620 6172 webserver.cc:460] Webserver started at http://127.0.98.2:33443/ using document root <none> and password file <none>
I20260928 02:34:50.589823 6172 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:50.589869 6172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:50.589982 6172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:50.590965 6172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-1/data/instance:
uuid: "acfe283b45ef4577b9609d5298e1d3eb"
format_stamp: "Formatted at 2026-09-28 02:34:50 on dist-test-slave-fz47"
server_key: "ec36b2bf43ce31bb4a0c4a348467969a"
server_key_iv: "efe3317c312ce97dc1a2c51b2f0c4727"
server_key_version: "encryptionkey@0"
I20260928 02:34:50.591310 6172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-1/wal/instance:
uuid: "acfe283b45ef4577b9609d5298e1d3eb"
format_stamp: "Formatted at 2026-09-28 02:34:50 on dist-test-slave-fz47"
server_key: "ec36b2bf43ce31bb4a0c4a348467969a"
server_key_iv: "efe3317c312ce97dc1a2c51b2f0c4727"
server_key_version: "encryptionkey@0"
I20260928 02:34:50.592479 6172 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:34:50.593077 6187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:50.593204 6172 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:50.593310 6172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "acfe283b45ef4577b9609d5298e1d3eb"
format_stamp: "Formatted at 2026-09-28 02:34:50 on dist-test-slave-fz47"
server_key: "ec36b2bf43ce31bb4a0c4a348467969a"
server_key_iv: "efe3317c312ce97dc1a2c51b2f0c4727"
server_key_version: "encryptionkey@0"
I20260928 02:34:50.593369 6172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:50.609069 6172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:50.610848 6172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:50.610983 6172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:50.611209 6172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:34:50.611497 6172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:50.611528 6172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:50.611557 6172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:50.611573 6172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:50.616815 6172 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40555
I20260928 02:34:50.616879 6300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40555 every 8 connection(s)
I20260928 02:34:50.617158 6172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 02:34:50 dist-test-slave-fz47 krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:50.625041 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 6172
I20260928 02:34:50.625151 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-1/wal/instance
I20260928 02:34:50.625358 392 external_mini_cluster.cc:1428] Setting key c61c989569e41b916026601eae4dbcb0
I20260928 02:34:50.625646 6016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:50.617848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44009 (local address 127.0.98.62:44483)
0928 02:34:50.617912 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:50.617914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:50.618086 (+ 172us) server_negotiation.cc:407] Connection header received
0928 02:34:50.618969 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:50.618970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:50.618980 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:50.619027 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:50.619574 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:50.620056 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:50.620949 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:50.621122 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:50.621164 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:50.623027 (+ 1863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:50.623029 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:50.623029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:50.623035 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:50.624369 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:50.624721 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:50.624723 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:50.624723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:50.624767 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:50.624945 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:50.624945 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:50.624946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:50.625070 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:50.625136 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:50.625504 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 02:34:50.625567 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 02:34:50.625844 6303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:50.617818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44483 (local address 127.0.98.2:44009)
0928 02:34:50.617974 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:50.618010 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:50.618887 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:50.619089 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:50.619093 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:50.619335 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:50.619516 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:50.619520 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:50.620118 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:50.620118 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:50.620893 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:50.620895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:50.621173 (+ 278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:50.621174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:50.621304 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:50.621711 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:50.621729 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:50.622920 (+ 1191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:50.624438 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:50.624444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:50.624445 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:50.624660 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:50.624809 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:50.624809 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:50.624809 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:50.624896 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:50.625151 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:50.625158 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:50.625347 (+ 189us) client_negotiation.cc:769] Sending connection context
0928 02:34:50.625477 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 02:34:50.625635 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260928 02:34:50.626241 6301 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44483
I20260928 02:34:50.626320 6301 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:50.626471 6301 heartbeater.cc:511] Master 127.0.98.62:44483 requested a full tablet report, sending...
I20260928 02:34:50.626801 5976 ts_manager.cc:194] Registered new tserver with Master: acfe283b45ef4577b9609d5298e1d3eb (127.0.98.2:40555)
I20260928 02:34:50.627249 5976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44009
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 02:34:50.647199 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44483
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:50.725078 6306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:50.725292 6306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:50.725333 6306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:50.726446 6306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:50.726480 6306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:50.726504 6306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:34:50.727887 6306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:50.728056 6306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:50.728221 6306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:50.732761 6312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:50.732815 6306 server_base.cc:1034] running on GCE node
W20260928 02:34:50.732888 6314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:50.732888 6311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:50.733196 6306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:50.733451 6306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:50.734578 6306 hybrid_clock.cc:648] HybridClock initialized: now 1790562890734565 us; error 28 us; skew 500 ppm
Sep 28 02:34:50 dist-test-slave-fz47 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:50.736840 6306 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 02:34:50.737290 6306 webserver.cc:460] Webserver started at http://127.0.98.3:44061/ using document root <none> and password file <none>
I20260928 02:34:50.737490 6306 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:50.737535 6306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:50.737653 6306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:50.738686 6306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-2/data/instance:
uuid: "8a88592c5d0c4890b13fde20c84fcf30"
format_stamp: "Formatted at 2026-09-28 02:34:50 on dist-test-slave-fz47"
server_key: "fbc09cb936e5855ead470049fc0568c7"
server_key_iv: "abf2a3e9c0e03db3f861dc6d3c2dfefb"
server_key_version: "encryptionkey@0"
I20260928 02:34:50.739004 6306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8a88592c5d0c4890b13fde20c84fcf30"
format_stamp: "Formatted at 2026-09-28 02:34:50 on dist-test-slave-fz47"
server_key: "fbc09cb936e5855ead470049fc0568c7"
server_key_iv: "abf2a3e9c0e03db3f861dc6d3c2dfefb"
server_key_version: "encryptionkey@0"
I20260928 02:34:50.740152 6306 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:34:50.740774 6321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:50.740916 6306 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:34:50.740991 6306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "8a88592c5d0c4890b13fde20c84fcf30"
format_stamp: "Formatted at 2026-09-28 02:34:50 on dist-test-slave-fz47"
server_key: "fbc09cb936e5855ead470049fc0568c7"
server_key_iv: "abf2a3e9c0e03db3f861dc6d3c2dfefb"
server_key_version: "encryptionkey@0"
I20260928 02:34:50.741055 6306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:50.747402 6306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:50.749317 6306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:50.749437 6306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:50.749620 6306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:34:50.749877 6306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:50.749907 6306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:50.749933 6306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:50.749956 6306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:50.755110 6306 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42225
I20260928 02:34:50.755146 6434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42225 every 8 connection(s)
I20260928 02:34:50.755416 6306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 02:34:50 dist-test-slave-fz47 krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:50.761127 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 6306
I20260928 02:34:50.761233 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/minicluster-data/ts-2/wal/instance
I20260928 02:34:50.761447 392 external_mini_cluster.cc:1428] Setting key d1eab6931ccfaf74876d2a63d62f42ed
I20260928 02:34:50.763831 6016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:50.756003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60019 (local address 127.0.98.62:44483)
0928 02:34:50.756078 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:50.756080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:50.756237 (+ 157us) server_negotiation.cc:407] Connection header received
0928 02:34:50.757089 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:50.757090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:50.757099 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:50.757146 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:50.757739 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:50.758210 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:50.759126 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:50.759315 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:50.759368 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:50.761615 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:50.761617 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:50.761617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:50.761626 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:50.762858 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:50.763171 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:50.763172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:50.763172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:50.763219 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:50.763373 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:50.763374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:50.763374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:50.763466 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:50.763509 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:50.763708 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 02:34:50.763770 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 02:34:50.763980 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:50.755986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44483 (local address 127.0.98.3:60019)
0928 02:34:50.756127 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:50.756159 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:50.757011 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:50.757200 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:50.757203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:50.757470 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:50.757680 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:50.757684 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:50.758270 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:50.758271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:50.759055 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:50.759058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:50.759380 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:50.759381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:50.759481 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:50.759990 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:50.759999 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:50.761531 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:50.762917 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:50.762922 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:50.762924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:50.763116 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:50.763255 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:50.763256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:50.763256 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:50.763332 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:50.763516 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:50.763519 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:50.763621 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 02:34:50.763695 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 02:34:50.763804 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20260928 02:34:50.764343 6435 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44483
I20260928 02:34:50.764424 6435 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:50.764577 6435 heartbeater.cc:511] Master 127.0.98.62:44483 requested a full tablet report, sending...
I20260928 02:34:50.764894 5976 ts_manager.cc:194] Registered new tserver with Master: 8a88592c5d0c4890b13fde20c84fcf30 (127.0.98.3:42225)
I20260928 02:34:50.765311 5976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60019
I20260928 02:34:50.772840 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/test-admin.keytab.
Sep 28 02:34:50 dist-test-slave-fz47 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:50.785305 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 02:34:50.786763 6442 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:34:50.786079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36426 (local address 127.0.98.1:40733)
0928 02:34:50.786238 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:50.786242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:50.786252 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:34:50.786275 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:50.786277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:50.786295 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:50.786356 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:50.786689 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36426: BlockingRecv error: recv got EOF from 127.0.0.1:36426 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
W20260928 02:34:50.786803 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/krb5cc' of type FILE
Sep 28 02:34:50 dist-test-slave-fz47 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:50.787647 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:34:50 dist-test-slave-fz47 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:50.788376 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:34:50 dist-test-slave-fz47 krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 02:34:50.795239 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:50.788797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36428 (local address 127.0.98.1:40733)
0928 02:34:50.788841 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:50.788843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:50.788898 (+ 55us) server_negotiation.cc:407] Connection header received
0928 02:34:50.788947 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:50.788948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:50.788957 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:50.788992 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:50.789333 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:50.790037 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:50.790682 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:50.790982 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:50.791031 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:50.792829 (+ 1798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:50.792836 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:50.792837 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:50.792843 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:50.794244 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:50.794553 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:50.794557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:50.794557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:50.794606 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:50.794762 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:50.794763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:50.794763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:50.794881 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:50.794997 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:50.795073 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 02:34:50.795138 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 02:34:50.796681 6442 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:34:50.796253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36430 (local address 127.0.98.1:40733)
0928 02:34:50.796306 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:50.796308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:50.796315 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:34:50.796375 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:50.796375 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:50.796384 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:50.796423 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:50.796645 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36430: BlockingRecv error: recv got EOF from 127.0.0.1:36430 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 02:34:50.796761 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/krb5cc)
Sep 28 02:34:50 dist-test-slave-fz47 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:50.797627 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:34:50 dist-test-slave-fz47 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:50.798316 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:34:50 dist-test-slave-fz47 krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 02:34:50.804174 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:50.798735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36438 (local address 127.0.98.1:40733)
0928 02:34:50.798784 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:50.798785 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:50.798793 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:34:50.798841 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:50.798841 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:50.798850 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:50.798889 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:50.799189 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:50.799645 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:50.800266 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:50.800483 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:50.800536 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:50.802268 (+ 1732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:50.802269 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:50.802270 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:50.802276 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:50.803270 (+ 994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:50.803548 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:50.803549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:50.803550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:50.803597 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:50.803762 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:50.803763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:50.803763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:50.803847 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:50.803923 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:50.804019 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 02:34:50.804089 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 02:34:50.805477 6442 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:34:50.804960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36448 (local address 127.0.98.1:40733)
0928 02:34:50.804991 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:50.804992 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:50.805029 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:34:50.805081 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:50.805081 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:50.805089 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:50.805130 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:50.805437 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36448: BlockingRecv error: recv got EOF from 127.0.0.1:36448 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 02:34:50.805538 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790562862804661-392-0/krb5kdc/krb5cc' of type FILE
Sep 28 02:34:50 dist-test-slave-fz47 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:50.806316 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:34:50 dist-test-slave-fz47 krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:50.806988 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:34:50 dist-test-slave-fz47 krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 02:34:50.812575 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:50.807390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36450 (local address 127.0.98.1:40733)
0928 02:34:50.807442 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:50.807443 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:50.807451 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:34:50.807491 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:50.807491 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:50.807498 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:50.807537 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:50.807798 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:50.808227 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:50.808815 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:50.808975 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:50.809012 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:50.810652 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:50.810653 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:50.810653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:50.810658 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:50.811839 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:50.812089 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:50.812089 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:50.812089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:50.812130 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:50.812255 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:50.812255 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:50.812256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:50.812326 (+ 70us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:50.812383 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:50.812454 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 02:34:50.812500 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 02:34:50.813128 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 6038
I20260928 02:34:50.817508 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 6172
I20260928 02:34:50.821894 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 6306
I20260928 02:34:50.825153 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 5944
2026-09-28T02:34:50Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1749 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:34:50 dist-test-slave-fz47 krb5kdc[6455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:34:50 dist-test-slave-fz47 krb5kdc[6455](info): set up 2 sockets
Sep 28 02:34:50 dist-test-slave-fz47 krb5kdc[6455](info): commencing operation
krb5kdc: starting...
W20260928 02:34:51.802621 392 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:34:51 dist-test-slave-fz47 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:34:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:34:51Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260928 02:34:51.915845 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34979
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34979
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:51.992970 6468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:51.993177 6468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:51.993198 6468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:51.994335 6468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:34:51.994375 6468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:51.994387 6468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:51.994411 6468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:34:51.994432 6468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:34:51.995853 6468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:51.996078 6468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:51.996284 6468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:52.000895 6474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:52.001009 6476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:52.001046 6473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:52.001047 6468 server_base.cc:1034] running on GCE node
I20260928 02:34:52.001582 6468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:52.001850 6468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:52.003006 6468 hybrid_clock.cc:648] HybridClock initialized: now 1790562892002988 us; error 28 us; skew 500 ppm
Sep 28 02:34:52 dist-test-slave-fz47 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562892, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:52.007926 6468 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260928 02:34:52.008369 6468 webserver.cc:460] Webserver started at http://127.0.98.62:46565/ using document root <none> and password file <none>
I20260928 02:34:52.008559 6468 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:52.008605 6468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:52.008723 6468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:52.009793 6468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "6d0b62d9ab8d41ffa729271c52ec57ba"
format_stamp: "Formatted at 2026-09-28 02:34:52 on dist-test-slave-fz47"
server_key: "9e8f6465b6b9abff4f44a401bcb51bc1"
server_key_iv: "f5830a8fd1d46ee453cbb56627838391"
server_key_version: "encryptionkey@0"
I20260928 02:34:52.010104 6468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "6d0b62d9ab8d41ffa729271c52ec57ba"
format_stamp: "Formatted at 2026-09-28 02:34:52 on dist-test-slave-fz47"
server_key: "9e8f6465b6b9abff4f44a401bcb51bc1"
server_key_iv: "f5830a8fd1d46ee453cbb56627838391"
server_key_version: "encryptionkey@0"
I20260928 02:34:52.011266 6468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:34:52.011876 6483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:52.012017 6468 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:52.012094 6468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "6d0b62d9ab8d41ffa729271c52ec57ba"
format_stamp: "Formatted at 2026-09-28 02:34:52 on dist-test-slave-fz47"
server_key: "9e8f6465b6b9abff4f44a401bcb51bc1"
server_key_iv: "f5830a8fd1d46ee453cbb56627838391"
server_key_version: "encryptionkey@0"
I20260928 02:34:52.012158 6468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:52.027168 6468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:52.028723 6468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:52.028849 6468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:52.032490 6468 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34979
I20260928 02:34:52.032531 6535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34979 every 8 connection(s)
I20260928 02:34:52.032827 6468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:34:52.033741 6536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:52.035877 6536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba: Bootstrap starting.
I20260928 02:34:52.036392 6536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:52.036646 6536 log.cc:824] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:52.037304 6536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba: No bootstrap required, opened a new log
I20260928 02:34:52.038836 6536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } }
I20260928 02:34:52.038964 6536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:52.038995 6536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d0b62d9ab8d41ffa729271c52ec57ba, State: Initialized, Role: FOLLOWER
I20260928 02:34:52.039134 6536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } }
I20260928 02:34:52.039203 6536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:34:52.039227 6536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:34:52.039271 6536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:52.040001 6536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } }
I20260928 02:34:52.040096 6536 leader_election.cc:302] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d0b62d9ab8d41ffa729271c52ec57ba; no voters:
I20260928 02:34:52.040243 6536 leader_election.cc:288] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:34:52.040237 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 6468
I20260928 02:34:52.040274 6539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:52.040328 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:34:52.040418 6539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [term 1 LEADER]: Becoming Leader. State: Replica: 6d0b62d9ab8d41ffa729271c52ec57ba, State: Running, Role: LEADER
I20260928 02:34:52.040477 6536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:34:52.040517 392 external_mini_cluster.cc:1428] Setting key b4a54e4f9c9381d5656e8e2b969f31eb
I20260928 02:34:52.040503 6539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } }
I20260928 02:34:52.040812 6539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d0b62d9ab8d41ffa729271c52ec57ba. Latest consensus state: current_term: 1 leader_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } }
I20260928 02:34:52.040957 6539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:52.041214 6549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:34:52.041296 6541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } }
I20260928 02:34:52.041584 6541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:52.042090 6549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:34:52.043569 6549 catalog_manager.cc:1347] Generated new cluster ID: ef1a356553cc4e3597de38b414bb0511
I20260928 02:34:52.043615 6549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:34:52 dist-test-slave-fz47 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 02:34:52.051491 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.041131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35026 (local address 127.0.98.62:34979)
0928 02:34:52.041419 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:52.041428 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:52.041462 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:34:52.041629 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:52.041638 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:52.041718 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:52.042619 (+ 901us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:52.042730 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.043900 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.044923 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.045519 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.045607 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:52.048164 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:52.048168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:52.048168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:52.048179 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:52.050026 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.050368 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.050371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.050371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.050482 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.050645 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.050645 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.050646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.050925 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:52.051079 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:52.051160 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 02:34:52.051288 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":83,"threads_started":1}
I20260928 02:34:52.059162 6549 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:34:52.059829 6549 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:34:52.076998 6549 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba: Generated new TSK 0
I20260928 02:34:52.077193 6549 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260928 02:34:52.111697 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34979
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:52.188726 6562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:52.188922 6562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:52.188952 6562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:52.190106 6562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:52.190146 6562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:52.190181 6562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:34:52.191563 6562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:52.191789 6562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:52.191978 6562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:52.196599 6567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:52.196599 6568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:52.196828 6562 server_base.cc:1034] running on GCE node
W20260928 02:34:52.196949 6570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:52.197145 6562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:52.197394 6562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:52.198541 6562 hybrid_clock.cc:648] HybridClock initialized: now 1790562892198527 us; error 30 us; skew 500 ppm
Sep 28 02:34:52 dist-test-slave-fz47 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562892, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:52.200773 6562 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 02:34:52.201146 6562 webserver.cc:460] Webserver started at http://127.0.98.1:38859/ using document root <none> and password file <none>
I20260928 02:34:52.201370 6562 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:52.201414 6562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:52.201516 6562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:52.202463 6562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-0/data/instance:
uuid: "2f784af0f7a0463583c819774b174e5c"
format_stamp: "Formatted at 2026-09-28 02:34:52 on dist-test-slave-fz47"
server_key: "3c36691032b3c1bc5c6624cc0899bbbf"
server_key_iv: "4c1d774f0077d0cfb28c83ef542ee775"
server_key_version: "encryptionkey@0"
I20260928 02:34:52.202768 6562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2f784af0f7a0463583c819774b174e5c"
format_stamp: "Formatted at 2026-09-28 02:34:52 on dist-test-slave-fz47"
server_key: "3c36691032b3c1bc5c6624cc0899bbbf"
server_key_iv: "4c1d774f0077d0cfb28c83ef542ee775"
server_key_version: "encryptionkey@0"
I20260928 02:34:52.203931 6562 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:34:52.204551 6577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:52.204694 6562 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:52.204762 6562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "2f784af0f7a0463583c819774b174e5c"
format_stamp: "Formatted at 2026-09-28 02:34:52 on dist-test-slave-fz47"
server_key: "3c36691032b3c1bc5c6624cc0899bbbf"
server_key_iv: "4c1d774f0077d0cfb28c83ef542ee775"
server_key_version: "encryptionkey@0"
I20260928 02:34:52.204823 6562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:52.234444 6562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:52.236186 6562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:52.236310 6562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:52.236526 6562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:34:52.236822 6562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:52.236857 6562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:52.236878 6562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:52.236889 6562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:52.242314 6562 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39961
I20260928 02:34:52.242383 6690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39961 every 8 connection(s)
I20260928 02:34:52.242657 6562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
I20260928 02:34:52.246065 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 6562
I20260928 02:34:52.246162 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-0/wal/instance
I20260928 02:34:52.246353 392 external_mini_cluster.cc:1428] Setting key 161c433a1899eb96764c0ee622b39195
Sep 28 02:34:52 dist-test-slave-fz47 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562892, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 02:34:52.251881 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.243320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32889 (local address 127.0.98.62:34979)
0928 02:34:52.243370 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:52.243372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:52.243576 (+ 204us) server_negotiation.cc:407] Connection header received
0928 02:34:52.244070 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:52.244071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:52.244086 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:52.244138 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:52.245098 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.245655 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.246971 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.247192 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.247245 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:52.249428 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:52.249430 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:52.249430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:52.249437 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:52.250671 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.251033 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.251034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.251034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.251085 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.251296 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.251297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.251297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.251402 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:52.251455 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:52.251713 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 02:34:52.251796 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 02:34:52.252045 6693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.243284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.98.1:32889)
0928 02:34:52.243460 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:52.243496 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:52.243967 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:52.244209 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:52.244214 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:52.244749 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:52.244968 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:52.245007 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.245735 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.245736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:52.246903 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:52.246906 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.247257 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.247258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:52.247395 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:52.247858 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:52.247867 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:52.249341 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:52.250742 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:52.250748 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:52.250750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:52.250956 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:52.251136 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:52.251137 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:52.251137 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:52.251242 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:52.251455 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:52.251461 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:52.251599 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 02:34:52.251684 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 02:34:52.251831 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":93,"threads_started":1}
I20260928 02:34:52.252485 6691 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34979
I20260928 02:34:52.252568 6691 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:52.252727 6691 heartbeater.cc:511] Master 127.0.98.62:34979 requested a full tablet report, sending...
I20260928 02:34:52.253201 6500 ts_manager.cc:194] Registered new tserver with Master: 2f784af0f7a0463583c819774b174e5c (127.0.98.1:39961)
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 02:34:52.253823 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:32889
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260928 02:34:52.267942 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34979
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:52.344046 6696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:52.344233 6696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:52.344264 6696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:52.345355 6696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:52.345391 6696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:52.345424 6696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:34:52.346800 6696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:52.347035 6696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:52.347239 6696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:52.351785 6702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:52.351778 6704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:52.351794 6701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:52.352078 6696 server_base.cc:1034] running on GCE node
I20260928 02:34:52.352272 6696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:52.352479 6696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:52.353612 6696 hybrid_clock.cc:648] HybridClock initialized: now 1790562892353597 us; error 26 us; skew 500 ppm
Sep 28 02:34:52 dist-test-slave-fz47 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562892, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:52.355890 6696 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 02:34:52.356320 6696 webserver.cc:460] Webserver started at http://127.0.98.2:35621/ using document root <none> and password file <none>
I20260928 02:34:52.356503 6696 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:52.356542 6696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:52.356628 6696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:52.357632 6696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/data/instance:
uuid: "f7e8161e260e47cbad68b8e313fe23fd"
format_stamp: "Formatted at 2026-09-28 02:34:52 on dist-test-slave-fz47"
server_key: "9dda8cda5653fb188980c95bde69b440"
server_key_iv: "666d412231f443979a40eab4be49729b"
server_key_version: "encryptionkey@0"
I20260928 02:34:52.357913 6696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f7e8161e260e47cbad68b8e313fe23fd"
format_stamp: "Formatted at 2026-09-28 02:34:52 on dist-test-slave-fz47"
server_key: "9dda8cda5653fb188980c95bde69b440"
server_key_iv: "666d412231f443979a40eab4be49729b"
server_key_version: "encryptionkey@0"
I20260928 02:34:52.359084 6696 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:34:52.359727 6711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:52.359884 6696 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:52.359961 6696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "f7e8161e260e47cbad68b8e313fe23fd"
format_stamp: "Formatted at 2026-09-28 02:34:52 on dist-test-slave-fz47"
server_key: "9dda8cda5653fb188980c95bde69b440"
server_key_iv: "666d412231f443979a40eab4be49729b"
server_key_version: "encryptionkey@0"
I20260928 02:34:52.360013 6696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:52.367955 6696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:52.369897 6696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:52.369997 6696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:52.370163 6696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:34:52.370404 6696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:52.370432 6696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:52.370451 6696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:52.370467 6696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:52.375679 6696 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46089
I20260928 02:34:52.375720 6824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46089 every 8 connection(s)
I20260928 02:34:52.375994 6696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 02:34:52 dist-test-slave-fz47 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562892, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 02:34:52.381703 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 6696
I20260928 02:34:52.381783 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/wal/instance
I20260928 02:34:52.381959 392 external_mini_cluster.cc:1428] Setting key b7f0a6f07c79d132a3aae371f4439e6a
I20260928 02:34:52.384611 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.376612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40981 (local address 127.0.98.62:34979)
0928 02:34:52.376690 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:52.376693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:52.376851 (+ 158us) server_negotiation.cc:407] Connection header received
0928 02:34:52.377326 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:52.377327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:52.377336 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:52.377383 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:52.378275 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.378787 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.379734 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.379917 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.379974 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:52.381965 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:52.381967 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:52.381967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:52.381976 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:52.383295 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.383719 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.383720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.383720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.383780 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.384036 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.384037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.384037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.384178 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:52.384240 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:52.384445 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 02:34:52.384527 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 02:34:52.384694 6827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.376582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.98.2:40981)
0928 02:34:52.376738 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:52.376773 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:52.377228 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:52.377443 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:52.377446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:52.377979 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:52.378166 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:52.378171 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.378863 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.378864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:52.379655 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:52.379657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.379978 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.379978 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:52.380073 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:52.380481 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:52.380488 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:52.381875 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:52.383369 (+ 1494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:52.383375 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:52.383377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:52.383642 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:52.383847 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:52.383847 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:52.383848 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:52.383980 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:52.384232 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:52.384235 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:52.384343 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 02:34:52.384416 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:34:52.384527 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20260928 02:34:52.385075 6825 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34979
I20260928 02:34:52.385171 6825 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:52.385378 6825 heartbeater.cc:511] Master 127.0.98.62:34979 requested a full tablet report, sending...
I20260928 02:34:52.385710 6500 ts_manager.cc:194] Registered new tserver with Master: f7e8161e260e47cbad68b8e313fe23fd (127.0.98.2:46089)
I20260928 02:34:52.386196 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40981
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260928 02:34:52.403726 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34979
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:52.481402 6830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:52.481590 6830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:52.481611 6830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:52.482684 6830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:52.482715 6830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:52.482738 6830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:34:52.484074 6830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:52.484239 6830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:52.484402 6830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:52.488884 6836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:52.488893 6835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:52.488899 6838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:52.488978 6830 server_base.cc:1034] running on GCE node
I20260928 02:34:52.489300 6830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:52.489497 6830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:52.490620 6830 hybrid_clock.cc:648] HybridClock initialized: now 1790562892490609 us; error 27 us; skew 500 ppm
Sep 28 02:34:52 dist-test-slave-fz47 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562892, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:52.492744 6830 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 02:34:52.493188 6830 webserver.cc:460] Webserver started at http://127.0.98.3:43423/ using document root <none> and password file <none>
I20260928 02:34:52.493400 6830 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:52.493441 6830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:52.493526 6830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:52.494427 6830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/data/instance:
uuid: "ced49025d91f44928924ee460747c5b4"
format_stamp: "Formatted at 2026-09-28 02:34:52 on dist-test-slave-fz47"
server_key: "5f61cc3fcc0337e1cd96af8cdcd694d1"
server_key_iv: "56d07d55db3061820ef5ba1393eccd1a"
server_key_version: "encryptionkey@0"
I20260928 02:34:52.494699 6830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ced49025d91f44928924ee460747c5b4"
format_stamp: "Formatted at 2026-09-28 02:34:52 on dist-test-slave-fz47"
server_key: "5f61cc3fcc0337e1cd96af8cdcd694d1"
server_key_iv: "56d07d55db3061820ef5ba1393eccd1a"
server_key_version: "encryptionkey@0"
I20260928 02:34:52.495782 6830 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:34:52.496373 6845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:52.496519 6830 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:52.496592 6830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "ced49025d91f44928924ee460747c5b4"
format_stamp: "Formatted at 2026-09-28 02:34:52 on dist-test-slave-fz47"
server_key: "5f61cc3fcc0337e1cd96af8cdcd694d1"
server_key_iv: "56d07d55db3061820ef5ba1393eccd1a"
server_key_version: "encryptionkey@0"
I20260928 02:34:52.496644 6830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:52.516789 6830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:52.518615 6830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:52.518790 6830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:52.518965 6830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:34:52.519243 6830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:52.519273 6830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:52.519302 6830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:52.519325 6830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:52.524590 6830 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33037
I20260928 02:34:52.524628 6958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33037 every 8 connection(s)
I20260928 02:34:52.524914 6830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
I20260928 02:34:52.527761 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 6830
I20260928 02:34:52.527844 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/wal/instance
I20260928 02:34:52.528002 392 external_mini_cluster.cc:1428] Setting key 754be615e6291dcbe7bc85a6f6fcbefb
Sep 28 02:34:52 dist-test-slave-fz47 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562892, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 02:34:52.533388 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.525589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44029 (local address 127.0.98.62:34979)
0928 02:34:52.525636 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:52.525638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:52.525828 (+ 190us) server_negotiation.cc:407] Connection header received
0928 02:34:52.526347 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:52.526348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:52.526360 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:52.526410 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:52.527335 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.527863 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.528981 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.529176 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.529229 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:52.531269 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:52.531270 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:52.531271 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:52.531276 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:52.532320 (+ 1044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.532676 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.532676 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.532677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.532720 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.532889 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.532889 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.532890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.532979 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:52.533020 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:52.533245 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 02:34:52.533328 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 02:34:52.533573 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.525557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.98.3:44029)
0928 02:34:52.525718 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:52.525752 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:52.526245 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:52.526489 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:52.526492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:52.527048 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:52.527236 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:52.527240 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.527967 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.527968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:52.528914 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:52.528917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.529236 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.529237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:52.529367 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:52.529865 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:52.529873 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:52.531198 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:52.532406 (+ 1208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:52.532411 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:52.532413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:52.532617 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:52.532763 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:52.532763 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:52.532764 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:52.532849 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:52.533018 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:52.533021 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:52.533144 (+ 123us) client_negotiation.cc:769] Sending connection context
0928 02:34:52.533223 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 02:34:52.533385 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":83,"threads_started":1}
I20260928 02:34:52.533957 6959 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34979
I20260928 02:34:52.534041 6959 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:52.534205 6959 heartbeater.cc:511] Master 127.0.98.62:34979 requested a full tablet report, sending...
I20260928 02:34:52.534508 6500 ts_manager.cc:194] Registered new tserver with Master: ced49025d91f44928924ee460747c5b4 (127.0.98.3:33037)
I20260928 02:34:52.534837 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44029
I20260928 02:34:52.539498 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260928 02:34:52.562072 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45279
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34979,127.0.98.61:45279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:52.639590 6964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:52.639829 6964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:52.639854 6964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:52.641013 6964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:34:52.641052 6964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:52.641064 6964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:52.641075 6964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:34:52.641088 6964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:34:52.642601 6964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34979,127.0.98.61:45279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
W20260928 02:34:52.642701 6964 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:34979,127.0.98.61:45279'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 02:34:52.642836 6964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:52.643028 6964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:52.647411 6969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:52.647610 6970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:52.647681 6964 server_base.cc:1034] running on GCE node
W20260928 02:34:52.647615 6972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:52.647933 6964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:52.648207 6964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:52.649312 6964 hybrid_clock.cc:648] HybridClock initialized: now 1790562892649291 us; error 31 us; skew 500 ppm
Sep 28 02:34:52 dist-test-slave-fz47 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562892, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:52.651533 6964 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260928 02:34:52.651937 6964 webserver.cc:460] Webserver started at http://127.0.98.61:44211/ using document root <none> and password file <none>
I20260928 02:34:52.652144 6964 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:52.652192 6964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:52.652310 6964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:52.653345 6964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-1/data/instance:
uuid: "37118a91babb433594ad029895cee4a7"
format_stamp: "Formatted at 2026-09-28 02:34:52 on dist-test-slave-fz47"
server_key: "cf758198b0bc3db87e8fb94264c12b33"
server_key_iv: "f72ee7d144a7348cb80e57c088de673a"
server_key_version: "encryptionkey@0"
I20260928 02:34:52.653666 6964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-1/wal/instance:
uuid: "37118a91babb433594ad029895cee4a7"
format_stamp: "Formatted at 2026-09-28 02:34:52 on dist-test-slave-fz47"
server_key: "cf758198b0bc3db87e8fb94264c12b33"
server_key_iv: "f72ee7d144a7348cb80e57c088de673a"
server_key_version: "encryptionkey@0"
I20260928 02:34:52.654832 6964 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:34:52.655527 6979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:52.655658 6964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:34:52.655727 6964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-1/wal
uuid: "37118a91babb433594ad029895cee4a7"
format_stamp: "Formatted at 2026-09-28 02:34:52 on dist-test-slave-fz47"
server_key: "cf758198b0bc3db87e8fb94264c12b33"
server_key_iv: "f72ee7d144a7348cb80e57c088de673a"
server_key_version: "encryptionkey@0"
I20260928 02:34:52.655797 6964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:52.688517 6964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:52.690593 6964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:52.690718 6964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:52.694288 6964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45279
I20260928 02:34:52.694324 7031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45279 every 8 connection(s)
I20260928 02:34:52.694617 6964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-1/data/info.pb
I20260928 02:34:52.695504 7032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:52.696501 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 6964
I20260928 02:34:52.697242 7032 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:34:52 dist-test-slave-fz47 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562892, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 28 02:34:52 dist-test-slave-fz47 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 02:34:52.705029 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.698198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35040 (local address 127.0.98.62:34979)
0928 02:34:52.698259 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:52.698261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:52.698407 (+ 146us) server_negotiation.cc:407] Connection header received
0928 02:34:52.698466 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:52.698467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:52.698478 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:52.698519 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:52.698965 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.699500 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.700419 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.700617 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.700669 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:52.702553 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:52.702554 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:52.702555 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:52.702561 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:52.703721 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.704089 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.704090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.704091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.704164 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.704370 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.704371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.704371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.704550 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:52.704620 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:52.704864 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 02:34:52.704938 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:34:52.705186 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.698132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.0.1:35040)
0928 02:34:52.698315 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:52.698349 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:52.698408 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:52.698555 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:52.698557 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:52.698793 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:52.698884 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:52.698888 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.699585 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.699586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:52.700317 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:52.700319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.700681 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.700681 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:52.700743 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:52.701165 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:52.701172 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:52.702479 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:52.703790 (+ 1311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:52.703795 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:52.703797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:52.704028 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:52.704217 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:52.704218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:52.704218 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:52.704317 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:52.704616 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:52.704620 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:52.704729 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 02:34:52.704839 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 02:34:52.704964 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":89,"threads_started":1}
I20260928 02:34:52.705693 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.697089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46888 (local address 127.0.98.61:45279)
0928 02:34:52.697230 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:52.697238 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:52.697281 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:34:52.697771 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:52.697778 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:52.697829 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:52.697964 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:52.698556 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.699447 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.700183 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.700485 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.700573 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:52.703035 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:52.703040 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:52.703041 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:52.703046 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:52.704412 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.704760 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.704763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.704763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.704820 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.704995 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.704995 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.704996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.705133 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:52.705289 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:52.705535 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 02:34:52.705606 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":58,"threads_started":1}
I20260928 02:34:52.705758 7032 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:34:52 dist-test-slave-fz47 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562892, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 02:34:52.710889 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.705988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45279 (local address 127.0.0.1:46898)
0928 02:34:52.706030 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:52.706039 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:52.706064 (+ 25us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:52.706171 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:52.706171 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:52.706286 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:52.706317 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:52.706318 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.706858 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.706859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:52.707332 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:52.707333 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.707643 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.707643 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:52.707695 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:52.708015 (+ 320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:52.708019 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:52.708989 (+ 970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:52.710088 (+ 1099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:52.710089 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:52.710089 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:52.710234 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:52.710377 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:52.710377 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:52.710377 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:52.710425 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:52.710604 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:52.710605 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:52.710638 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:34:52.710695 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 02:34:52.710779 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 02:34:52.710945 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.706020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46898 (local address 127.0.98.61:45279)
0928 02:34:52.706050 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:52.706052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:52.706075 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:34:52.706113 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:52.706114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:52.706122 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:52.706167 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:52.706367 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.706794 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.707396 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.707590 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.707645 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:52.709055 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:52.709056 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:52.709056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:52.709062 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:52.710025 (+ 963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.710288 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.710288 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.710289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.710329 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.710471 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.710472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.710472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.710555 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:52.710605 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:52.710718 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 02:34:52.710824 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":16}
I20260928 02:34:52.712021 7032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7: Bootstrap starting.
I20260928 02:34:52.712762 7032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:52.712997 7032 log.cc:824] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:52.713683 7032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7: No bootstrap required, opened a new log
I20260928 02:34:52.715000 7032 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } }
I20260928 02:34:52.715116 7032 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:52.715152 7032 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37118a91babb433594ad029895cee4a7, State: Initialized, Role: FOLLOWER
I20260928 02:34:52.715266 7032 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } }
I20260928 02:34:52.715464 7038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } } }
I20260928 02:34:52.715525 7032 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:34:52.715543 7038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 02:34:52.716920 7049 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:34:52.716967 7049 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:34:52.718348 6964 master_runner.cc:428] Detected that this master 37118a91babb433594ad029895cee4a7 is joining an existing cluster
I20260928 02:34:52.718384 6964 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:45279
I20260928 02:34:52.725013 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.719188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45279 (local address 127.0.0.1:46908)
0928 02:34:52.719438 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:52.719446 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:52.719482 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:52.719583 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:52.719584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:52.720072 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:52.720128 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:52.720129 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.720942 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.720943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:52.721621 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:52.721623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.721977 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.721978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:52.722028 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:52.722347 (+ 319us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:52.722351 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:52.722666 (+ 315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:52.724064 (+ 1398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:52.724065 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:52.724065 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:52.724212 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:52.724366 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:52.724367 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:52.724367 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:52.724429 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:52.724690 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:52.724690 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:52.724729 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 02:34:52.724794 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 02:34:52.724886 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":90,"threads_started":1}
I20260928 02:34:52.725069 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.719227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46908 (local address 127.0.98.61:45279)
0928 02:34:52.719259 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:52.719260 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:52.719488 (+ 228us) server_negotiation.cc:407] Connection header received
0928 02:34:52.719524 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:52.719525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:52.719532 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:52.719574 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:52.720205 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.720864 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.721679 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.721918 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.721976 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:52.722713 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:52.722714 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:52.722714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:52.722719 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:52.723997 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.724264 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.724265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.724265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.724310 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.724493 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.724494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.724495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.724623 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:52.724693 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:52.724824 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 02:34:52.724943 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":18}
I20260928 02:34:52.725356 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.719389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35048 (local address 127.0.98.62:34979)
0928 02:34:52.719429 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:52.719431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:52.719438 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:34:52.719457 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:52.719458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:52.719466 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:52.719519 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:52.719912 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.720492 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.721379 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.721669 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.721743 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:52.722661 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:52.722662 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:52.722662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:52.722669 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:52.724280 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.724596 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.724597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.724598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.724650 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.724826 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.724827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.724827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.724941 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:52.724981 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:52.725185 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 02:34:52.725273 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:34:52.725628 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.719187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.0.1:35048)
0928 02:34:52.719318 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:52.719328 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:52.719373 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:52.719560 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:52.719561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:52.719754 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:52.719833 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:52.719835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.720552 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.720553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:52.721241 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:52.721243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.721737 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.721737 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:52.721794 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:52.722198 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:52.722201 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:52.722570 (+ 369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:52.724353 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:52.724353 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:52.724354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:52.724533 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:52.724710 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:52.724711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:52.724711 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:52.724760 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:52.725025 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:52.725025 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:52.725070 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 02:34:52.725427 (+ 357us) client_negotiation.cc:240] Negotiation successful
0928 02:34:52.725520 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":95,"threads_started":1}
Sep 28 02:34:52 dist-test-slave-fz47 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562892, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 02:34:52.733724 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.727561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46912 (local address 127.0.98.61:45279)
0928 02:34:52.727597 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:52.727598 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:52.727706 (+ 108us) server_negotiation.cc:407] Connection header received
0928 02:34:52.727758 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:52.727759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:52.727768 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:52.727828 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:52.728149 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.728826 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.729541 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.729828 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.729892 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:52.731631 (+ 1739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:52.731632 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:52.731632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:52.731637 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:52.732719 (+ 1082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.733109 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.733109 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.733110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.733150 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.733341 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.733342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.733342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.733423 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:52.733466 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:52.733630 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 02:34:52.733675 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:34:52.733821 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.727506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45279 (local address 127.0.0.1:46912)
0928 02:34:52.727614 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:52.727650 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:52.727710 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:52.727832 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:52.727834 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:52.728047 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:52.728088 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:52.728089 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.728889 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.728890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:52.729482 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:52.729484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.729888 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.729889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:52.729944 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:52.730291 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:52.730296 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:52.731556 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:52.732786 (+ 1230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:52.732791 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:52.732793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:52.733035 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:52.733201 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:52.733202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:52.733202 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:52.733286 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:52.733476 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:52.733481 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:52.733530 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 02:34:52.733606 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 02:34:52.733713 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":61,"threads_started":1}
I20260928 02:34:52.734318 6500 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:45279: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "37118a91babb433594ad029895cee4a7"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 45279
}
attrs {
promote: true
}
}
dest_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba"
cas_config_opid_index: -1
I20260928 02:34:52.734613 6500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: true } }
I20260928 02:34:52.735030 7059 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 37118a91babb433594ad029895cee4a7 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: true } } }
I20260928 02:34:52.735294 7006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:52.735952 7006 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [term 1 FOLLOWER]: Refusing update from remote peer 6d0b62d9ab8d41ffa729271c52ec57ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 02:34:52.735981 7059 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:45279
I20260928 02:34:52.736186 6964 master.cc:537] Master@127.0.98.61:45279 shutting down...
I20260928 02:34:52.736233 7059 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:52.736625 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: true } } }
I20260928 02:34:52.736698 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [sys.catalog]: This master's current role is: LEADER
W20260928 02:34:52.736819 6487 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 02:34:52.737376 6487 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba -> Peer 37118a91babb433594ad029895cee4a7 (127.0.98.61:45279): Couldn't send request to peer 37118a91babb433594ad029895cee4a7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 02:34:52.738678 6964 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:34:52.738756 6964 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:52.738803 6964 tablet_replica.cc:330] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7: stopping tablet replica
I20260928 02:34:52.750926 6964 master.cc:552] Master@127.0.98.61:45279 shutdown complete.
I20260928 02:34:52.750988 6964 master_runner.cc:305] Clearing existing system tablet
I20260928 02:34:52.751103 6964 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:52.751171 6964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:52.751802 6964 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:52.752249 7064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:52.752357 6964 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:52.752411 6964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:52.752434 6964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-1/wal
uuid: "37118a91babb433594ad029895cee4a7"
format_stamp: "Formatted at 2026-09-28 02:34:52 on dist-test-slave-fz47"
server_key: "cf758198b0bc3db87e8fb94264c12b33"
server_key_iv: "f72ee7d144a7348cb80e57c088de673a"
server_key_version: "encryptionkey@0"
I20260928 02:34:52.753206 6964 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:52.754582 6964 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 02:34:52.754648 6964 log.cc:1196] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 02:34:52.754820 6964 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7: Deleting consensus metadata
I20260928 02:34:52.754930 6964 master_runner.cc:315] Copying system tablet from 127.0.98.62:34979
I20260928 02:34:52.755498 6964 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:34979
I20260928 02:34:52.761350 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.755727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.0.1:35060)
0928 02:34:52.755838 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:52.755849 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:52.755926 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:52.756033 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:52.756033 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:52.756221 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:52.756305 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:52.756307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.757044 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.757045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:52.758006 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:52.758009 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.758311 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.758311 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:52.758376 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:52.758750 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:52.758755 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:52.759106 (+ 351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:52.760411 (+ 1305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:52.760412 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:52.760413 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:52.760620 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:52.760773 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:52.760773 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:52.760774 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:52.760832 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:52.761013 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:52.761014 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:52.761056 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 02:34:52.761128 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 02:34:52.761222 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":71,"threads_started":1}
I20260928 02:34:52.761332 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.755805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35060 (local address 127.0.98.62:34979)
0928 02:34:52.755849 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:52.755851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:52.755909 (+ 58us) server_negotiation.cc:407] Connection header received
0928 02:34:52.755975 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:52.755976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:52.755984 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:52.756027 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:52.756399 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.756976 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.758082 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.758253 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.758298 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:52.759184 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:52.759185 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:52.759186 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:52.759191 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:52.760340 (+ 1149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.760687 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.760687 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.760688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.760730 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.760881 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.760881 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.760881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.760964 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:52.761005 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:52.761150 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 02:34:52.761216 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:34:52.761883 6520 tablet_copy_service.cc:140] P 6d0b62d9ab8d41ffa729271c52ec57ba: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 37118a91babb433594ad029895cee4a7 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:35060)
I20260928 02:34:52.761971 6520 tablet_copy_service.cc:161] P 6d0b62d9ab8d41ffa729271c52ec57ba: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 37118a91babb433594ad029895cee4a7 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:35060: session id = 37118a91babb433594ad029895cee4a7-00000000000000000000000000000000
I20260928 02:34:52.762593 6520 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 02:34:52.763170 6964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 02:34:52.764011 6964 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7: tablet copy: Starting download of 0 data blocks...
I20260928 02:34:52.764134 6964 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7: tablet copy: Starting download of 1 WAL segments...
I20260928 02:34:52.764853 6964 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 02:34:52.765565 6520 tablet_copy_service.cc:342] P 6d0b62d9ab8d41ffa729271c52ec57ba: Request end of tablet copy session 37118a91babb433594ad029895cee4a7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:35060
I20260928 02:34:52.765614 6520 tablet_copy_service.cc:433] P 6d0b62d9ab8d41ffa729271c52ec57ba: ending tablet copy session 37118a91babb433594ad029895cee4a7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 37118a91babb433594ad029895cee4a7
I20260928 02:34:52.767798 6964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:52.768767 7071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:52.768848 7072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:52.768963 6964 server_base.cc:1034] running on GCE node
W20260928 02:34:52.769030 7074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:52.769150 6964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:52.769294 6964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:52.770411 6964 hybrid_clock.cc:648] HybridClock initialized: now 1790562892770404 us; error 22 us; skew 500 ppm
Sep 28 02:34:52 dist-test-slave-fz47 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562892, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:52.771601 6964 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260928 02:34:52.771840 6964 webserver.cc:460] Webserver started at http://127.0.98.61:39851/ using document root <none> and password file <none>
I20260928 02:34:52.771945 6964 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:52.771978 6964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:52.772596 6964 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:34:52.773066 7081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:52.773183 6964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:52.773231 6964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-1/wal
uuid: "37118a91babb433594ad029895cee4a7"
format_stamp: "Formatted at 2026-09-28 02:34:52 on dist-test-slave-fz47"
server_key: "cf758198b0bc3db87e8fb94264c12b33"
server_key_iv: "f72ee7d144a7348cb80e57c088de673a"
server_key_version: "encryptionkey@0"
I20260928 02:34:52.773348 6964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:52.782759 6964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:52.784356 6964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:52.788218 6964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45279
I20260928 02:34:52.788443 7133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45279 every 8 connection(s)
I20260928 02:34:52.788568 6964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-1/data/info.pb
I20260928 02:34:52.789191 7134 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:34:52.789608 7134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7: Bootstrap starting.
I20260928 02:34:52.791602 7134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:34:52.791776 7134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7: Bootstrap complete.
I20260928 02:34:52.791929 7134 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: true } }
I20260928 02:34:52.791990 7134 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 37118a91babb433594ad029895cee4a7, State: Initialized, Role: LEARNER
I20260928 02:34:52.792022 7134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: true } }
I20260928 02:34:52.792176 7134 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:34:52.792168 7137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: true } } }
I20260928 02:34:52.792258 7137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:34:52.793457 7148 catalog_manager.cc:1259] Loaded cluster ID: ef1a356553cc4e3597de38b414bb0511
I20260928 02:34:52.793561 7148 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7: loading cluster ID for follower catalog manager: success
I20260928 02:34:52.795035 7148 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7: acquiring CA information for follower catalog manager: success
I20260928 02:34:52.795341 7148 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:34:52.813491 7149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.806996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46944 (local address 127.0.98.61:45279)
0928 02:34:52.807096 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:52.807098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:52.807108 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:34:52.807143 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:52.807144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:52.807156 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:52.807236 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:52.807763 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.808792 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.809562 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.809867 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.809943 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:52.810968 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:52.810970 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:52.810970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:52.810977 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:52.812449 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.812742 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.812743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.812744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.812804 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.812955 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.812956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.812956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.813080 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:52.813209 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:52.813302 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 02:34:52.813379 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":51,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260928 02:34:52.835678 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41171
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34979,127.0.98.61:45279,127.0.98.60:41171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:52.913197 7152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:52.913425 7152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:52.913446 7152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:52.914556 7152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:34:52.914597 7152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:52.914609 7152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:52.914620 7152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:34:52.914631 7152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:34:52.916438 7152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34979,127.0.98.61:45279,127.0.98.60:41171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:52.916684 7152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:52.916880 7152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:52.921300 7157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:52.921334 7160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:52.921337 7158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:52.921453 7152 server_base.cc:1034] running on GCE node
I20260928 02:34:52.921655 7152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:52.921901 7152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:52.923024 7152 hybrid_clock.cc:648] HybridClock initialized: now 1790562892923009 us; error 31 us; skew 500 ppm
Sep 28 02:34:52 dist-test-slave-fz47 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562892, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:52.925217 7152 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260928 02:34:52.925606 7152 webserver.cc:460] Webserver started at http://127.0.98.60:43895/ using document root <none> and password file <none>
I20260928 02:34:52.925774 7152 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:52.925810 7152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:52.925895 7152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:52.926810 7152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-2/data/instance:
uuid: "c6b3724cda7546ca9a2c651f5c1a0688"
format_stamp: "Formatted at 2026-09-28 02:34:52 on dist-test-slave-fz47"
server_key: "801292ada4a84f5f01898dcd07bb0508"
server_key_iv: "9463187ea3c1df4ae6988e002a7c7831"
server_key_version: "encryptionkey@0"
I20260928 02:34:52.927088 7152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-2/wal/instance:
uuid: "c6b3724cda7546ca9a2c651f5c1a0688"
format_stamp: "Formatted at 2026-09-28 02:34:52 on dist-test-slave-fz47"
server_key: "801292ada4a84f5f01898dcd07bb0508"
server_key_iv: "9463187ea3c1df4ae6988e002a7c7831"
server_key_version: "encryptionkey@0"
I20260928 02:34:52.928153 7152 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 02:34:52.928731 7167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:52.928866 7152 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:52.928941 7152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-2/wal
uuid: "c6b3724cda7546ca9a2c651f5c1a0688"
format_stamp: "Formatted at 2026-09-28 02:34:52 on dist-test-slave-fz47"
server_key: "801292ada4a84f5f01898dcd07bb0508"
server_key_iv: "9463187ea3c1df4ae6988e002a7c7831"
server_key_version: "encryptionkey@0"
I20260928 02:34:52.928988 7152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:52.942343 7152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:52.943888 7152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:52.944003 7152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:52.948104 7152 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41171
I20260928 02:34:52.948132 7219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41171 every 8 connection(s)
I20260928 02:34:52.948432 7152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-2/data/info.pb
I20260928 02:34:52.949331 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 7152
I20260928 02:34:52.949451 7220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:52.951577 7220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:34:52 dist-test-slave-fz47 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 28 02:34:52 dist-test-slave-fz47 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562892, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 02:34:52.958715 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.952208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35062 (local address 127.0.98.62:34979)
0928 02:34:52.952269 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:52.952272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:52.952280 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:34:52.952335 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:52.952336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:52.952347 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:52.952389 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:52.952802 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.953291 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.954125 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.954302 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.954354 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:52.956241 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:52.956242 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:52.956242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:52.956249 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:52.957595 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.957911 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.957912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.957913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.957960 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.958176 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.958178 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.958178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.958290 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:52.958328 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:52.958557 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 02:34:52.958627 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 02:34:52.958870 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.952054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.0.1:35062)
0928 02:34:52.952186 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:52.952221 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:52.952276 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:52.952410 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:52.952412 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:52.952659 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:52.952725 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:52.952727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.953368 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.953369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:52.954050 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:52.954053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.954365 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.954365 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:52.954429 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:52.954787 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:52.954794 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:52.956164 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:52.957659 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:52.957663 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:52.957665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:52.957833 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:52.958029 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:52.958030 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:52.958030 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:52.958105 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:52.958371 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:52.958375 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:52.958428 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 02:34:52.958517 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 02:34:52.958706 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":61,"thread_start_us":64,"threads_started":1}
I20260928 02:34:52.958931 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.949871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60918 (local address 127.0.98.60:41171)
0928 02:34:52.950028 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:52.950035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:52.950057 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:34:52.950474 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:52.950480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:52.950523 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:52.950627 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:52.951332 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.952354 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.953050 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.953517 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.953609 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:52.955755 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:52.955761 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:52.955761 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:52.955767 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:52.957520 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.957883 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.957886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.957887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.957968 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.958129 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.958130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.958130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.958364 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:52.958489 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:52.958556 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 02:34:52.958654 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20260928 02:34:52.959488 7220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:34:52 dist-test-slave-fz47 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562892, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 02:34:52.964622 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.959705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45279 (local address 127.0.0.1:46948)
0928 02:34:52.959728 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:52.959738 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:52.959777 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:52.959891 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:52.959892 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:52.959983 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:52.960023 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:52.960024 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.960556 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.960557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:52.961024 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:52.961025 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.961410 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.961411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:52.961456 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:52.961714 (+ 258us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:52.961717 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:52.962737 (+ 1020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:52.963967 (+ 1230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:52.963968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:52.963968 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:52.964107 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:52.964237 (+ 130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:52.964238 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:52.964238 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:52.964276 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:52.964429 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:52.964429 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:52.964458 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 02:34:52.964500 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 02:34:52.964555 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13}
I20260928 02:34:52.964633 7149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.959775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46948 (local address 127.0.98.61:45279)
0928 02:34:52.959813 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:52.959815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:52.959821 (+ 6us) server_negotiation.cc:407] Connection header received
0928 02:34:52.959841 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:52.959842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:52.959851 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:52.959889 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:52.960062 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.960509 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.961104 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.961348 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.961411 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:52.962804 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:52.962805 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:52.962805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:52.962811 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:52.963905 (+ 1094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.964159 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.964159 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.964160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.964203 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.964312 (+ 109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.964312 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.964313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.964398 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:52.964438 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:52.964517 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 02:34:52.964575 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:34:52.964958 7220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41171 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:34:52 dist-test-slave-fz47 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562892, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 02:34:52.969975 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.965153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41171 (local address 127.0.0.1:60932)
0928 02:34:52.965190 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:52.965199 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:52.965231 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:52.965341 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:52.965342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:52.965445 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:52.965471 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:52.965472 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.966009 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.966010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:52.966451 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:52.966452 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.966760 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.966761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:52.966802 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:52.967083 (+ 281us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:52.967086 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:52.968020 (+ 934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:52.969122 (+ 1102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:52.969122 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:52.969123 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:52.969289 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:52.969433 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:52.969434 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:52.969434 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:52.969482 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:52.969668 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:52.969668 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:52.969701 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:34:52.969758 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 02:34:52.969857 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 02:34:52.970031 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.965200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60932 (local address 127.0.98.60:41171)
0928 02:34:52.965239 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:52.965241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:52.965249 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:34:52.965296 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:52.965297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:52.965304 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:52.965341 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:52.965526 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.965952 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.966509 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.966709 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.966758 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:52.968088 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:52.968089 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:52.968089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:52.968095 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:52.969044 (+ 949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.969344 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.969345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.969345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.969388 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.969530 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.969530 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.969530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.969619 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:52.969668 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:52.969781 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 02:34:52.969922 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":26}
I20260928 02:34:52.971092 7220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688: Bootstrap starting.
I20260928 02:34:52.971792 7220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:52.971992 7220 log.cc:824] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:52.972569 7220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688: No bootstrap required, opened a new log
I20260928 02:34:52.973927 7220 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41171 } }
I20260928 02:34:52.974037 7220 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:52.974059 7220 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b3724cda7546ca9a2c651f5c1a0688, State: Initialized, Role: FOLLOWER
I20260928 02:34:52.974169 7220 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41171 } }
I20260928 02:34:52.974435 7220 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:34:52.974388 7225 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41171 } } }
I20260928 02:34:52.974488 7225 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 02:34:52.975791 7236 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:34:52.975842 7236 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:34:52.977880 7152 master_runner.cc:428] Detected that this master c6b3724cda7546ca9a2c651f5c1a0688 is joining an existing cluster
I20260928 02:34:52.977922 7152 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:41171
I20260928 02:34:52.985566 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.979423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45279 (local address 127.0.0.1:46962)
0928 02:34:52.979597 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:52.979606 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:52.979664 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:52.979809 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:52.979809 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:52.979962 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:52.980016 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:52.980017 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.980872 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.980873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:52.981936 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:52.981939 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.982368 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.982369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:52.982429 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:52.982750 (+ 321us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:52.982754 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:52.983003 (+ 249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:52.984518 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:52.984519 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:52.984520 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:52.984851 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:52.985048 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:52.985048 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:52.985049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:52.985091 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:52.985340 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:52.985341 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:52.985379 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:34:52.985426 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 02:34:52.985496 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":63,"threads_started":1}
I20260928 02:34:52.985638 7149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.979348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46962 (local address 127.0.98.61:45279)
0928 02:34:52.979383 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:52.979385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:52.979643 (+ 258us) server_negotiation.cc:407] Connection header received
0928 02:34:52.979723 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:52.979724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:52.979733 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:52.979757 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:52.980082 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.980791 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.982009 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.982296 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.982370 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:52.983073 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:52.983075 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:52.983075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:52.983081 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:52.984432 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.984926 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.984927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.984928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.984987 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:52.985127 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:52.985127 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:52.985128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:52.985243 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:52.985342 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:52.985469 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 02:34:52.985540 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 02:34:53.003913 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.979117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60948 (local address 127.0.98.60:41171)
0928 02:34:52.979146 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:52.979147 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:52.979153 (+ 6us) server_negotiation.cc:407] Connection header received
0928 02:34:52.979168 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:52.979168 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:52.979175 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:52.979212 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:52.979536 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.980182 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.981100 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.981400 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.981449 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:52.982316 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:52.982317 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:52.982317 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:52.982323 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:53.002749 (+ 20426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.003082 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.003083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.003083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.003143 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.003346 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.003347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.003347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.003483 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:53.003546 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:53.003703 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 02:34:53.003791 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":16}
I20260928 02:34:53.004006 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.978991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41171 (local address 127.0.0.1:60948)
0928 02:34:52.979080 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:52.979088 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:52.979123 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:52.979224 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:52.979225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:52.979419 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:52.979482 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:52.979484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.980248 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.980248 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:52.981022 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:52.981023 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.981498 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.981498 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:52.981548 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:52.981922 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:52.981926 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:52.982249 (+ 323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:53.002818 (+ 20569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:53.002819 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:53.002820 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:53.003006 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:53.003211 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:53.003211 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:53.003212 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:53.003287 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:53.003551 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:53.003552 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:53.003603 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 02:34:53.003675 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 02:34:53.003849 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":72,"thread_start_us":60,"threads_started":1}
I20260928 02:34:53.005250 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.978784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.0.1:35070)
0928 02:34:52.978902 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:52.978912 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:52.978973 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:52.979143 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:52.979144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:52.979345 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:52.979422 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:52.979423 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.980338 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.980339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:52.981195 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:52.981197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.981720 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.981721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:52.981780 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.001840 (+ 20060us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:53.001846 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:53.002351 (+ 505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:53.004283 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:53.004284 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:53.004284 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:53.004463 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:53.004665 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:53.004665 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:53.004666 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:53.004731 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:53.004944 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:53.004945 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:53.004996 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 02:34:53.005054 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 02:34:53.005149 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":1,"thread_start_us":74,"threads_started":1}
I20260928 02:34:53.005244 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:52.978849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35070 (local address 127.0.98.62:34979)
0928 02:34:52.978896 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:52.978898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:52.978964 (+ 66us) server_negotiation.cc:407] Connection header received
0928 02:34:52.979034 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:52.979035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:52.979044 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:52.979099 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:52.979511 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.980248 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.981313 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:52.981610 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:52.981648 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.002490 (+ 20842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:53.002492 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:53.002492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:53.002501 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:53.004023 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.004550 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.004551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.004552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.004608 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.004782 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.004783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.004784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.004892 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:53.004945 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:53.005082 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 02:34:53.005158 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 02:34:53.010128 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 6562
Sep 28 02:34:53 dist-test-slave-fz47 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562892, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 02:34:53.013445 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.007392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60954 (local address 127.0.98.60:41171)
0928 02:34:53.007423 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:53.007425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:53.007432 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:34:53.007487 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:53.007487 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:53.007495 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:53.007542 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:53.007817 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.008275 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.009223 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.009459 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.009509 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.011350 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:53.011351 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:53.011352 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:53.011357 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:53.012513 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.012790 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.012791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.012791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.012839 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.012992 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.012992 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.012993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.013087 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:53.013138 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:53.013285 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 02:34:53.013359 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 02:34:53.013449 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.007341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41171 (local address 127.0.0.1:60954)
0928 02:34:53.007388 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:53.007398 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:53.007430 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:53.007545 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:53.007546 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:53.007708 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:53.007754 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.007756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.008337 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.008338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.009147 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.009149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.009522 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.009523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.009592 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.009997 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:53.010002 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:53.011280 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:53.012583 (+ 1303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:53.012584 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:53.012584 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:53.012733 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:53.012892 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:53.012893 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:53.012893 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:53.012944 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:53.013137 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:53.013138 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:53.013180 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 02:34:53.013238 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 02:34:53.013360 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 02:34:53.014153 6500 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:41171: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 41171
}
attrs {
promote: true
}
}
dest_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba"
cas_config_opid_index: 5
I20260928 02:34:53.014381 6500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: true } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: true } }
I20260928 02:34:53.014696 7058 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c6b3724cda7546ca9a2c651f5c1a0688 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: true } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: true } } }
I20260928 02:34:53.014914 7194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:53.015684 7058 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:41171
I20260928 02:34:53.015743 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: true } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: true } } }
I20260928 02:34:53.015801 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:53.015955 7194 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [term 1 FOLLOWER]: Refusing update from remote peer 6d0b62d9ab8d41ffa729271c52ec57ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 02:34:53.016239 7060 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:53.016607 7152 master.cc:537] Master@127.0.98.60:41171 shutting down...
I20260928 02:34:53.017421 7194 raft_consensus.cc:858] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41171 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: true } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: true } }
W20260928 02:34:53.017478 7194 raft_consensus.cc:1565] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41171 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: true } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: true } }
I20260928 02:34:53.017958 7194 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 37118a91babb433594ad029895cee4a7 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: true } } }
I20260928 02:34:53.019104 7248 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: true } } }
I20260928 02:34:53.019304 7248 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 02:34:53.019364 7060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: true } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: true } } }
I20260928 02:34:53.019435 7060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:53.019812 7245 mvcc.cc:205] Tried to move back new op lower bound from 7334145605946089472 to 7334145605798219776. Current Snapshot: MvccSnapshot[applied={T|T < 7334145605807632384}]
W20260928 02:34:53.019896 6485 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba -> Peer c6b3724cda7546ca9a2c651f5c1a0688 (127.0.98.60:41171): Couldn't send request to peer c6b3724cda7546ca9a2c651f5c1a0688. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 02:34:53.021034 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39961
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34979
--webserver_port=38859
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34979,127.0.98.61:45279
--tserver_master_addrs=127.0.98.62:34979,127.0.98.61:45279,127.0.98.60:41171 with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:34:53.022300 7225 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d0b62d9ab8d41ffa729271c52ec57ba. Latest consensus state: current_term: 1 leader_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: true } } }
I20260928 02:34:53.022372 7225 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 02:34:53.023195 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.016254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45279 (local address 127.0.0.1:46968)
0928 02:34:53.016286 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:53.016296 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:53.016335 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:53.019612 (+ 3277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:53.019613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:53.019615 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:53.019653 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.019655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.020863 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.020864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.022070 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.022073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.022931 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.022932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.023002 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.023026 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:34:53.023081 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 02:34:53.023129 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 02:34:53.023445 7152 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260928 02:34:53.023475 7149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.016390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46968 (local address 127.0.98.61:45279)
0928 02:34:53.016428 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:53.016429 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:53.016437 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:34:53.016460 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:53.016460 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:53.016481 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:53.016538 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:53.019983 (+ 3445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.020742 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.022199 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.022849 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.022922 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.022995 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:53.023380 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 02:34:53.023419 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:34:53.023545 7152 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260928 02:34:53.023664 7247 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: true } } }
I20260928 02:34:53.023870 7247 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260928 02:34:53.023932 7152 tablet_replica.cc:330] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688: stopping tablet replica
I20260928 02:34:53.023916 7108 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [term 1 LEARNER]: Refusing update from remote peer 6d0b62d9ab8d41ffa729271c52ec57ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 02:34:53.024098 7058 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 02:34:53.024668 7108 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c6b3724cda7546ca9a2c651f5c1a0688 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: true } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: true } } }
I20260928 02:34:53.025923 7137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d0b62d9ab8d41ffa729271c52ec57ba. Latest consensus state: current_term: 1 leader_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: true } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: true } } }
I20260928 02:34:53.026000 7137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:34:53.026041 7137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: true } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: true } } }
I20260928 02:34:53.026110 7137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:34:53.029772 7060 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba: attempting to promote NON_VOTER 37118a91babb433594ad029895cee4a7 to VOTER
I20260928 02:34:53.029973 7060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: false } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: true } }
I20260928 02:34:53.030400 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.030300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41171 (local address 127.0.0.1:60970)
0928 02:34:53.030333 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:53.030373 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 02:34:53.030462 7108 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [term 1 LEARNER]: Refusing update from remote peer 6d0b62d9ab8d41ffa729271c52ec57ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 02:34:53.030735 7059 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 02:34:53.030879 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.030795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41171 (local address 127.0.0.1:60972)
0928 02:34:53.030832 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:53.030854 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 02:34:53.030956 6485 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba -> Peer c6b3724cda7546ca9a2c651f5c1a0688 (127.0.98.60:41171): Couldn't send request to peer c6b3724cda7546ca9a2c651f5c1a0688. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 02:34:53.031316 7060 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 37118a91babb433594ad029895cee4a7 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: false } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: true } } }
I20260928 02:34:53.032159 7060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: false } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: true } } }
I20260928 02:34:53.032230 7060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:53.032282 7060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: false } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: true } } }
I20260928 02:34:53.032328 7060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:53.032548 7108 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 37118a91babb433594ad029895cee4a7 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: false } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: true } } }
I20260928 02:34:53.033449 7254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: false } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: true } } }
I20260928 02:34:53.033545 7254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:34:53.037097 7152 master.cc:552] Master@127.0.98.60:41171 shutdown complete.
I20260928 02:34:53.037145 7152 master_runner.cc:305] Clearing existing system tablet
I20260928 02:34:53.037308 7152 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:53.037352 7152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:53.038012 7152 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:34:53.038502 7256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:53.038638 7152 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:53.038691 7152 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:53.038712 7152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-2/wal
uuid: "c6b3724cda7546ca9a2c651f5c1a0688"
format_stamp: "Formatted at 2026-09-28 02:34:52 on dist-test-slave-fz47"
server_key: "801292ada4a84f5f01898dcd07bb0508"
server_key_iv: "9463187ea3c1df4ae6988e002a7c7831"
server_key_version: "encryptionkey@0"
I20260928 02:34:53.039372 7152 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:53.040607 7152 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 02:34:53.040733 7152 log.cc:1196] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 02:34:53.040951 7152 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688: Deleting consensus metadata
I20260928 02:34:53.041055 7152 master_runner.cc:315] Copying system tablet from 127.0.98.62:34979
I20260928 02:34:53.041642 7152 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:34979
I20260928 02:34:53.048355 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.041917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.0.1:35072)
0928 02:34:53.042039 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:53.042050 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:53.042104 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:53.042224 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:53.042225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:53.042408 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:53.042469 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.042471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.043387 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.043388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.044173 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.044176 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.044607 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.044608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.044673 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.045132 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:53.045137 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:53.045582 (+ 445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:53.047323 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:53.047324 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:53.047324 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:53.047535 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:53.047740 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:53.047741 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:53.047741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:53.047801 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:53.048054 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:53.048054 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:53.048098 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 02:34:53.048163 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 02:34:53.048258 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":68,"threads_started":1}
I20260928 02:34:53.048362 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.042064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35072 (local address 127.0.98.62:34979)
0928 02:34:53.042100 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:53.042102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:53.042111 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:34:53.042158 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:53.042159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:53.042168 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:53.042222 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:53.042566 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.043302 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.044260 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.044528 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.044604 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.045669 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:53.045671 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:53.045671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:53.045678 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:53.047238 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.047611 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.047612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.047612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.047674 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.047860 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.047861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.047861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.047990 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:53.048061 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:53.048202 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 02:34:53.048277 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:34:53.048743 6520 tablet_copy_service.cc:140] P 6d0b62d9ab8d41ffa729271c52ec57ba: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c6b3724cda7546ca9a2c651f5c1a0688 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:35072)
I20260928 02:34:53.048794 6520 tablet_copy_service.cc:161] P 6d0b62d9ab8d41ffa729271c52ec57ba: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c6b3724cda7546ca9a2c651f5c1a0688 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:35072: session id = c6b3724cda7546ca9a2c651f5c1a0688-00000000000000000000000000000000
I20260928 02:34:53.049331 6520 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 02:34:53.049750 7152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 02:34:53.050802 7152 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688: tablet copy: Starting download of 0 data blocks...
I20260928 02:34:53.050931 7152 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688: tablet copy: Starting download of 1 WAL segments...
I20260928 02:34:53.051642 7152 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 02:34:53.052420 6520 tablet_copy_service.cc:342] P 6d0b62d9ab8d41ffa729271c52ec57ba: Request end of tablet copy session c6b3724cda7546ca9a2c651f5c1a0688-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:35072
I20260928 02:34:53.052476 6520 tablet_copy_service.cc:433] P 6d0b62d9ab8d41ffa729271c52ec57ba: ending tablet copy session c6b3724cda7546ca9a2c651f5c1a0688-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c6b3724cda7546ca9a2c651f5c1a0688
I20260928 02:34:53.054991 7152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:53.056110 7263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:53.056303 7266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:53.056427 7264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:53.056627 7152 server_base.cc:1034] running on GCE node
I20260928 02:34:53.056713 7152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:53.056833 7152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:53.064159 7152 hybrid_clock.cc:648] HybridClock initialized: now 1790562893064138 us; error 40 us; skew 500 ppm
Sep 28 02:34:53 dist-test-slave-fz47 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562893, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:53.065769 7152 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260928 02:34:53.066015 7152 webserver.cc:460] Webserver started at http://127.0.98.60:39549/ using document root <none> and password file <none>
I20260928 02:34:53.066123 7152 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:53.066154 7152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:53.066830 7152 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:53.067360 7273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:53.067469 7152 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:53.067509 7152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-2/wal
uuid: "c6b3724cda7546ca9a2c651f5c1a0688"
format_stamp: "Formatted at 2026-09-28 02:34:52 on dist-test-slave-fz47"
server_key: "801292ada4a84f5f01898dcd07bb0508"
server_key_iv: "9463187ea3c1df4ae6988e002a7c7831"
server_key_version: "encryptionkey@0"
I20260928 02:34:53.067574 7152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:53.085209 7152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:53.086967 7152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:53.089895 7152 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41171
I20260928 02:34:53.089982 7325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41171 every 8 connection(s)
I20260928 02:34:53.090205 7152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/master-2/data/info.pb
I20260928 02:34:53.091136 7326 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:34:53.091621 7326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688: Bootstrap starting.
I20260928 02:34:53.094074 7326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:34:53.094261 7326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688: Bootstrap complete.
I20260928 02:34:53.094451 7326 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: false } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: true } }
I20260928 02:34:53.094538 7326 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6b3724cda7546ca9a2c651f5c1a0688, State: Initialized, Role: LEARNER
I20260928 02:34:53.094589 7326 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: false } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: true } }
I20260928 02:34:53.094753 7326 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:34:53.094761 7329 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: false } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: true } } }
I20260928 02:34:53.094980 7329 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:34:53.096202 7340 catalog_manager.cc:1259] Loaded cluster ID: ef1a356553cc4e3597de38b414bb0511
I20260928 02:34:53.096276 7340 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688: loading cluster ID for follower catalog manager: success
I20260928 02:34:53.097473 7340 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688: acquiring CA information for follower catalog manager: success
I20260928 02:34:53.097826 7340 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 02:34:53.112149 7252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:53.112327 7252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:53.112356 7252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:53.113472 7252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:53.113511 7252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:53.113546 7252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:34:53.115018 7252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39961
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38859
--tserver_master_addrs=127.0.98.62:34979,127.0.98.61:45279,127.0.98.60:41171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:53.115242 7252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:53.115454 7252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:53.120820 7346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:53.120913 7252 server_base.cc:1034] running on GCE node
W20260928 02:34:53.120769 7345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:53.120760 7348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:53.121166 7252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:53.121407 7252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:53.122553 7252 hybrid_clock.cc:648] HybridClock initialized: now 1790562893122545 us; error 25 us; skew 500 ppm
Sep 28 02:34:53 dist-test-slave-fz47 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562893, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:53.124780 7252 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 02:34:53.125185 7252 webserver.cc:460] Webserver started at http://127.0.98.1:38859/ using document root <none> and password file <none>
I20260928 02:34:53.125406 7252 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:53.125458 7252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:53.126658 7252 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:34:53.127178 7355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:53.127316 7252 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:53.127391 7252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "2f784af0f7a0463583c819774b174e5c"
format_stamp: "Formatted at 2026-09-28 02:34:52 on dist-test-slave-fz47"
server_key: "3c36691032b3c1bc5c6624cc0899bbbf"
server_key_iv: "4c1d774f0077d0cfb28c83ef542ee775"
server_key_version: "encryptionkey@0"
I20260928 02:34:53.127709 7252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:53.149506 7252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:53.151623 7252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:53.151754 7252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:53.151957 7252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:34:53.152240 7252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:53.152272 7252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:53.152299 7252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:53.152318 7252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:53.157991 7252 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39961
I20260928 02:34:53.158041 7468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39961 every 8 connection(s)
I20260928 02:34:53.158356 7252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 02:34:53 dist-test-slave-fz47 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562893, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 28 02:34:53 dist-test-slave-fz47 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562893, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 28 02:34:53 dist-test-slave-fz47 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562893, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 02:34:53.166779 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 7252
I20260928 02:34:53.166926 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 6696
I20260928 02:34:53.169365 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.159219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44031 (local address 127.0.98.60:41171)
0928 02:34:53.159313 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:53.159315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:53.159428 (+ 113us) server_negotiation.cc:407] Connection header received
0928 02:34:53.160004 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:53.160006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:53.160018 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:53.160085 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:53.160618 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.161243 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.162882 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.163174 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.163229 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.166501 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:53.166503 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:53.166503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:53.166510 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:53.168002 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.168418 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.168419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.168420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.168474 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.168722 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.168723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.168723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.168839 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:53.168894 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:53.169151 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 02:34:53.169240 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":49,"threads_started":1}
I20260928 02:34:53.169696 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.159016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41171 (local address 127.0.98.1:44031)
0928 02:34:53.159254 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:53.159297 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:53.159798 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:53.160108 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:53.160109 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:53.160333 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:53.160511 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.160517 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.161842 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.161843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.162807 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.162809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.163229 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.163230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.163291 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.163859 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:53.163863 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:53.166187 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:53.168084 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:53.168091 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:53.168093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:53.168348 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:53.168530 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:53.168531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:53.168531 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:53.168655 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:53.168893 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:53.168899 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:53.169031 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 02:34:53.169121 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 02:34:53.169298 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":1,"thread_start_us":67,"threads_started":1}
I20260928 02:34:53.170027 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.159086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42721 (local address 127.0.98.62:34979)
0928 02:34:53.159136 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:53.159137 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:53.159410 (+ 273us) server_negotiation.cc:407] Connection header received
0928 02:34:53.159882 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:53.159883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:53.159895 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:53.159944 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:53.160604 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.161380 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.163014 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.163256 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.163319 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.167239 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:53.167241 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:53.167241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:53.167463 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:53.168951 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.169324 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.169325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.169326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.169378 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.169542 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.169543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.169543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.169654 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:53.169690 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:53.169856 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 02:34:53.169938 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 02:34:53.170296 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.159016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.98.1:42721)
0928 02:34:53.159288 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:53.159299 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:53.159786 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:53.160007 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:53.160012 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:53.160322 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:53.160510 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.160514 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.162220 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.162221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.162955 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.162957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.163325 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.163326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.163392 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.164017 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:53.164021 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:53.167138 (+ 3117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:53.169039 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:53.169039 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:53.169040 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:53.169225 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:53.169439 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:53.169439 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:53.169440 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:53.169488 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:53.169725 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:53.169726 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:53.169764 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:34:53.170108 (+ 344us) client_negotiation.cc:240] Negotiation successful
0928 02:34:53.170196 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":4,"thread_start_us":75,"threads_started":1}
I20260928 02:34:53.170703 7469 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34979
I20260928 02:34:53.170791 7469 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:53.170964 7469 heartbeater.cc:511] Master 127.0.98.62:34979 requested a full tablet report, sending...
I20260928 02:34:53.171350 6500 ts_manager.cc:194] Re-registered known tserver with Master: 2f784af0f7a0463583c819774b174e5c (127.0.98.1:39961)
I20260928 02:34:53.171732 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42721
I20260928 02:34:53.172760 7470 heartbeater.cc:347] Connected to a master server at 127.0.98.60:41171
I20260928 02:34:53.172808 7470 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:53.172812 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46089
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34979
--webserver_port=35621
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34979,127.0.98.61:45279
--tserver_master_addrs=127.0.98.62:34979,127.0.98.61:45279,127.0.98.60:41171 with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:34:53.172875 7470 heartbeater.cc:511] Master 127.0.98.60:41171 requested a full tablet report, sending...
I20260928 02:34:53.173223 7290 ts_manager.cc:194] Registered new tserver with Master: 2f784af0f7a0463583c819774b174e5c (127.0.98.1:39961)
I20260928 02:34:53.173743 7149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.159092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44793 (local address 127.0.98.61:45279)
0928 02:34:53.159143 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:53.159144 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:53.159389 (+ 245us) server_negotiation.cc:407] Connection header received
0928 02:34:53.159936 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:53.159937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:53.159946 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:53.159988 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:53.160624 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.161387 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.162769 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.163061 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.163130 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.165808 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:53.165810 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:53.165810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:53.165816 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:53.167321 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.172872 (+ 5551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.172874 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.172874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.172933 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.173123 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.173124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.173124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.173283 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:53.173327 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:53.173586 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 02:34:53.173656 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 02:34:53.173758 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.159016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45279 (local address 127.0.98.1:44793)
0928 02:34:53.159301 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:53.159308 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:53.159795 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:53.160361 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:53.160361 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:53.160483 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:53.160529 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.160530 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.161506 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.161507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.162690 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.162692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.163137 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.163138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.163272 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.163827 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:53.163836 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:53.165708 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:53.172578 (+ 6870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:53.172579 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:53.172579 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:53.172787 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:53.173004 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:53.173004 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:53.173005 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:53.173060 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:53.173372 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:53.173373 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:53.173429 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 02:34:53.173543 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 02:34:53.173646 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":180,"thread_start_us":62,"threads_started":1}
I20260928 02:34:53.174019 7471 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45279
I20260928 02:34:53.174065 7471 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:53.174113 7471 heartbeater.cc:511] Master 127.0.98.61:45279 requested a full tablet report, sending...
I20260928 02:34:53.174398 7098 ts_manager.cc:194] Registered new tserver with Master: 2f784af0f7a0463583c819774b174e5c (127.0.98.1:39961)
W20260928 02:34:53.251284 7477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:53.251500 7477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:53.251531 7477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:53.252683 7477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:53.252723 7477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:53.252758 7477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:34:53.254235 7477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46089
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35621
--tserver_master_addrs=127.0.98.62:34979,127.0.98.61:45279,127.0.98.60:41171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:53.254478 7477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:53.254686 7477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:53.259308 7482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:53.259357 7483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:53.259358 7485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:53.259496 7477 server_base.cc:1034] running on GCE node
I20260928 02:34:53.259752 7477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:53.259969 7477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:53.261113 7477 hybrid_clock.cc:648] HybridClock initialized: now 1790562893261113 us; error 38 us; skew 500 ppm
Sep 28 02:34:53 dist-test-slave-fz47 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562893, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:53.263403 7477 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 02:34:53.263801 7477 webserver.cc:460] Webserver started at http://127.0.98.2:35621/ using document root <none> and password file <none>
I20260928 02:34:53.263996 7477 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:53.264057 7477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:53.265367 7477 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:34:53.265906 7492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:53.266053 7477 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:53.266130 7477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "f7e8161e260e47cbad68b8e313fe23fd"
format_stamp: "Formatted at 2026-09-28 02:34:52 on dist-test-slave-fz47"
server_key: "9dda8cda5653fb188980c95bde69b440"
server_key_iv: "666d412231f443979a40eab4be49729b"
server_key_version: "encryptionkey@0"
I20260928 02:34:53.266494 7477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:53.287237 7477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:53.289052 7477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:53.289177 7477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:53.289405 7477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:34:53.289695 7477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:53.289728 7477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:53.289755 7477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:53.289778 7477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:53.295420 7477 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46089
I20260928 02:34:53.295460 7605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46089 every 8 connection(s)
I20260928 02:34:53.295804 7477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
I20260928 02:34:53.297281 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 7477
I20260928 02:34:53.297436 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 6830
Sep 28 02:34:53 dist-test-slave-fz47 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562893, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 28 02:34:53 dist-test-slave-fz47 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562893, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 02:34:53.302811 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33037
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34979
--webserver_port=43423
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34979,127.0.98.61:45279
--tserver_master_addrs=127.0.98.62:34979,127.0.98.61:45279,127.0.98.60:41171 with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:34:53.306241 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.296586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57923 (local address 127.0.98.60:41171)
0928 02:34:53.296628 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:53.296629 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:53.296859 (+ 230us) server_negotiation.cc:407] Connection header received
0928 02:34:53.297474 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:53.297475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:53.297489 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:53.297517 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:53.298402 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.299116 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.300152 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.300387 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.300449 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.303203 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:53.303204 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:53.303204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:53.303211 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:53.304897 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.305227 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.305228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.305228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.305299 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.305540 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.305541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.305541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.305663 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:53.305703 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:53.306079 (+ 376us) server_negotiation.cc:299] Negotiation successful
0928 02:34:53.306153 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:34:53.306241 7149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.296506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41565 (local address 127.0.98.61:45279)
0928 02:34:53.296611 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:53.296613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:53.296859 (+ 246us) server_negotiation.cc:407] Connection header received
0928 02:34:53.297397 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:53.297399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:53.297410 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:53.297439 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:53.298395 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.299125 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.300302 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.300566 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.300619 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.302625 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:53.302627 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:53.302627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:53.302634 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:53.304717 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.305198 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.305199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.305200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.305276 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.305551 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.305552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.305552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.305663 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:53.305703 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:53.306070 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 02:34:53.306147 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
Sep 28 02:34:53 dist-test-slave-fz47 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562893, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 02:34:53.306797 7612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.296494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45279 (local address 127.0.98.2:41565)
0928 02:34:53.296794 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:53.296804 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:53.297283 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:53.297930 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:53.297931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:53.298140 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:53.298298 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.298303 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.299232 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.299233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.300227 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.300229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.300628 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.300629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.300691 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.301157 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:53.301161 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:53.302507 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:53.304804 (+ 2297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:53.304811 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:53.304814 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:53.305115 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:53.305346 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:53.305346 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:53.305346 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:53.305468 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:53.305748 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:53.305755 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:53.305923 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 02:34:53.306356 (+ 433us) client_negotiation.cc:240] Negotiation successful
0928 02:34:53.306549 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":63,"thread_start_us":68,"threads_started":1}
I20260928 02:34:53.306799 7611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.296494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41171 (local address 127.0.98.2:57923)
0928 02:34:53.296804 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:53.296811 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:53.297286 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:53.297637 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:53.297642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:53.298016 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:53.298298 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.298313 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.299234 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.299235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.300075 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.300077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.300459 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.300460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.300585 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.301040 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:53.301048 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:53.303117 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:53.304965 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:53.304966 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:53.304966 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:53.305157 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:53.305365 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:53.305365 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:53.305366 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:53.305468 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:53.305748 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:53.305755 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:53.305923 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 02:34:53.306356 (+ 433us) client_negotiation.cc:240] Negotiation successful
0928 02:34:53.306600 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":303,"thread_start_us":58,"threads_started":1}
I20260928 02:34:53.307368 7608 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45279
I20260928 02:34:53.307462 7608 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:53.307639 7608 heartbeater.cc:511] Master 127.0.98.61:45279 requested a full tablet report, sending...
I20260928 02:34:53.307966 7607 heartbeater.cc:347] Connected to a master server at 127.0.98.60:41171
I20260928 02:34:53.308012 7607 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:53.308081 7607 heartbeater.cc:511] Master 127.0.98.60:41171 requested a full tablet report, sending...
I20260928 02:34:53.308120 7098 ts_manager.cc:194] Registered new tserver with Master: f7e8161e260e47cbad68b8e313fe23fd (127.0.98.2:46089)
I20260928 02:34:53.308336 7290 ts_manager.cc:194] Registered new tserver with Master: f7e8161e260e47cbad68b8e313fe23fd (127.0.98.2:46089)
I20260928 02:34:53.309432 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.296563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55879 (local address 127.0.98.62:34979)
0928 02:34:53.296641 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:53.296643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:53.296841 (+ 198us) server_negotiation.cc:407] Connection header received
0928 02:34:53.297398 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:53.297399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:53.297410 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:53.297439 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:53.302903 (+ 5464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.303610 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.304436 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.304714 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.304770 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.306955 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:53.306957 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:53.306957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:53.306964 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:53.308446 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.308784 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.308785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.308785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.308833 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.309001 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.309002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.309002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.309092 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:53.309134 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:53.309310 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 02:34:53.309374 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:34:53.309659 7610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.296494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.98.2:55879)
0928 02:34:53.296731 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:53.296759 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:53.297281 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:53.302585 (+ 5304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:53.302586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:53.302735 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:53.302790 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.302792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.303685 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.303686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.304363 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.304366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.304821 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.304822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.304880 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.305276 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:53.305280 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:53.306877 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:53.308515 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:53.308516 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:53.308517 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:53.308713 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:53.308884 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:53.308885 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:53.308886 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:53.308946 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:53.309141 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:53.309142 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:53.309185 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 02:34:53.309243 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 02:34:53.309353 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":78,"threads_started":1}
I20260928 02:34:53.309998 7606 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34979
I20260928 02:34:53.310050 7606 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:53.310099 7606 heartbeater.cc:511] Master 127.0.98.62:34979 requested a full tablet report, sending...
I20260928 02:34:53.310297 6500 ts_manager.cc:194] Re-registered known tserver with Master: f7e8161e260e47cbad68b8e313fe23fd (127.0.98.2:46089)
I20260928 02:34:53.310639 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55879
W20260928 02:34:53.384851 7613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:53.385062 7613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:53.385094 7613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:53.386294 7613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:53.386337 7613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:53.386372 7613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:34:53.387760 7613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43423
--tserver_master_addrs=127.0.98.62:34979,127.0.98.61:45279,127.0.98.60:41171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:53.387972 7613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:53.388166 7613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:53.392792 7619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:53.392820 7621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:53.392812 7618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:53.392932 7613 server_base.cc:1034] running on GCE node
I20260928 02:34:53.393216 7613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:53.393438 7613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:53.394570 7613 hybrid_clock.cc:648] HybridClock initialized: now 1790562893394562 us; error 35 us; skew 500 ppm
Sep 28 02:34:53 dist-test-slave-fz47 krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562893, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:53.396875 7613 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 02:34:53.397248 7613 webserver.cc:460] Webserver started at http://127.0.98.3:43423/ using document root <none> and password file <none>
I20260928 02:34:53.397459 7613 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:53.397516 7613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:53.398718 7613 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:34:53.399160 7628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:53.399300 7613 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:53.399374 7613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "ced49025d91f44928924ee460747c5b4"
format_stamp: "Formatted at 2026-09-28 02:34:52 on dist-test-slave-fz47"
server_key: "5f61cc3fcc0337e1cd96af8cdcd694d1"
server_key_iv: "56d07d55db3061820ef5ba1393eccd1a"
server_key_version: "encryptionkey@0"
I20260928 02:34:53.399675 7613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:53.416785 7613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:53.418435 7613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:53.418568 7613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:53.418752 7613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:34:53.419014 7613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:53.419047 7613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:53.419075 7613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:53.419098 7613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:53.424196 7613 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33037
I20260928 02:34:53.424260 7741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33037 every 8 connection(s)
I20260928 02:34:53.424540 7613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
I20260928 02:34:53.428156 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 7613
Sep 28 02:34:53 dist-test-slave-fz47 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562893, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 28 02:34:53 dist-test-slave-fz47 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562893, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 28 02:34:53 dist-test-slave-fz47 krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562893, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 02:34:53.435027 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.425165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41171 (local address 127.0.98.3:57727)
0928 02:34:53.425529 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:53.425537 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:53.426052 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:53.426304 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:53.426308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:53.426647 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:53.426822 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.426826 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.427733 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.427734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.428660 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.428662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.429419 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.429420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.429477 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.429959 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:53.429963 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:53.431644 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:53.433533 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:53.433539 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:53.433541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:53.433804 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:53.433997 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:53.433997 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:53.433998 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:53.434115 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:53.434385 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:53.434391 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:53.434550 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 02:34:53.434654 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 02:34:53.434798 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":38,"thread_start_us":86,"threads_started":1}
I20260928 02:34:53.435432 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.425468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57727 (local address 127.0.98.60:41171)
0928 02:34:53.425535 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:53.425536 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:53.425578 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:34:53.426212 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:53.426213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:53.426223 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:53.426269 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:53.426900 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.427655 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.428738 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.429026 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.429068 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.431723 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:53.431724 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:53.431724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:53.431731 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:53.433444 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.433879 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.433880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.433881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.433934 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.434183 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.434184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.434185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.434307 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:53.434362 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:53.435280 (+ 918us) server_negotiation.cc:299] Negotiation successful
0928 02:34:53.435349 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:34:53.435724 7743 heartbeater.cc:347] Connected to a master server at 127.0.98.60:41171
I20260928 02:34:53.435810 7743 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:53.435986 7743 heartbeater.cc:511] Master 127.0.98.60:41171 requested a full tablet report, sending...
I20260928 02:34:53.436404 7290 ts_manager.cc:194] Registered new tserver with Master: ced49025d91f44928924ee460747c5b4 (127.0.98.3:33037)
I20260928 02:34:53.436925 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.425410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44991 (local address 127.0.98.62:34979)
0928 02:34:53.425454 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:53.425456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:53.425577 (+ 121us) server_negotiation.cc:407] Connection header received
0928 02:34:53.426321 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:53.426322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:53.426331 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:53.426390 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:53.426923 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.427397 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.428721 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.429023 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.429062 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.432956 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:53.432957 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:53.432957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:53.432964 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:53.435026 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.435373 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.435374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.435374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.435426 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.435841 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.435842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.435843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.435961 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:53.436015 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:53.436771 (+ 756us) server_negotiation.cc:299] Negotiation successful
0928 02:34:53.436844 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:34:53.437034 7149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.425280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42029 (local address 127.0.98.61:45279)
0928 02:34:53.425325 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:53.425327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:53.425823 (+ 496us) server_negotiation.cc:407] Connection header received
0928 02:34:53.426135 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:53.426136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:53.426147 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:53.426186 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:53.426933 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.427669 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.428729 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.429014 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.429057 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.432607 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:53.432609 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:53.432609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:53.432615 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:53.434359 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.436272 (+ 1913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.436273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.436273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.436328 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.436536 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.436537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.436537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.436657 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:53.436696 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:53.436886 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 02:34:53.436952 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:34:53.437220 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.425165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.98.3:44991)
0928 02:34:53.425427 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:53.425490 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:53.426045 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:53.426398 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:53.426399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:53.426659 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:53.426823 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.426829 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.427487 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.427487 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.428625 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.428627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.429142 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.429143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.429300 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.429843 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:53.429852 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:53.432876 (+ 3024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:53.435109 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:53.435110 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:53.435111 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:53.435295 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:53.435491 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:53.435491 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:53.435492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:53.435767 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:53.436060 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:53.436061 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:53.436098 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 02:34:53.437035 (+ 937us) client_negotiation.cc:240] Negotiation successful
0928 02:34:53.437119 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":6,"thread_start_us":99,"threads_started":1}
I20260928 02:34:53.437330 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.425165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45279 (local address 127.0.98.3:42029)
0928 02:34:53.425765 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:53.425773 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:53.426051 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:53.426549 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:53.426550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:53.426691 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:53.426823 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.426829 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.427747 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.427748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.428628 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.428631 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.429142 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.429143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.429299 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.430094 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:53.430099 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:53.432499 (+ 2400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:53.435551 (+ 3052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:53.435551 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:53.435552 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:53.436196 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:53.436391 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:53.436391 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:53.436392 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:53.436454 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:53.436738 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:53.436739 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:53.436783 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 02:34:53.437093 (+ 310us) client_negotiation.cc:240] Negotiation successful
0928 02:34:53.437169 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":94,"thread_start_us":97,"threads_started":1}
I20260928 02:34:53.437685 7742 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34979
I20260928 02:34:53.437721 7742 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:53.437779 7742 heartbeater.cc:511] Master 127.0.98.62:34979 requested a full tablet report, sending...
I20260928 02:34:53.437876 7744 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45279
I20260928 02:34:53.437920 7744 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:53.437934 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.429481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35076 (local address 127.0.98.62:34979)
0928 02:34:53.429621 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:53.429623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:53.430151 (+ 528us) server_negotiation.cc:407] Connection header received
0928 02:34:53.430172 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:53.430172 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:53.430183 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:53.430237 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:53.430915 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.431609 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.433021 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.433720 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.433975 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.435264 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:53.435265 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:53.435265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:53.435271 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:53.436794 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.437144 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.437145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.437145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.437201 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.437396 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.437397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.437397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.437512 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:53.437604 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:53.437776 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 02:34:53.437852 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20260928 02:34:53.437978 7744 heartbeater.cc:511] Master 127.0.98.61:45279 requested a full tablet report, sending...
I20260928 02:34:53.437985 6500 ts_manager.cc:194] Re-registered known tserver with Master: ced49025d91f44928924ee460747c5b4 (127.0.98.3:33037)
I20260928 02:34:53.438225 7098 ts_manager.cc:194] Registered new tserver with Master: ced49025d91f44928924ee460747c5b4 (127.0.98.3:33037)
I20260928 02:34:53.438906 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44991
I20260928 02:34:53.442308 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.429846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60982 (local address 127.0.98.60:41171)
0928 02:34:53.430968 (+ 1122us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:53.430970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:53.431099 (+ 129us) server_negotiation.cc:407] Connection header received
0928 02:34:53.431116 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:53.431117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:53.431126 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:53.431174 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:53.434294 (+ 3120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.434996 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.438391 (+ 3395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.438671 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.438725 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.439961 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:53.439962 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:53.439963 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:53.439969 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:53.441123 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.441539 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.441540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.441540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.441632 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.441790 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.441791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.441791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.441913 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:53.441999 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:53.442127 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 02:34:53.442202 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1105,"thread_start_us":44,"threads_started":1}
I20260928 02:34:53.442318 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.429348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46978 (local address 127.0.98.61:45279)
0928 02:34:53.429601 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:53.429602 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:53.429795 (+ 193us) server_negotiation.cc:407] Connection header received
0928 02:34:53.429917 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:53.429918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:53.430585 (+ 667us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:53.430627 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:53.431861 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.432640 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.436121 (+ 3481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.436406 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.436494 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.439690 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:53.439691 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:53.439691 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:53.439698 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:53.441122 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.441610 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.441611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.441612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.441671 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.441826 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.441827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.441827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.441926 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:53.442004 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:53.442160 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 02:34:53.442230 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":47,"threads_started":1}
I20260928 02:34:53.439844 6500 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:34:53.442751 6500 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:34:53.449834 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.446253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41856 (local address 127.0.98.1:39961)
0928 02:34:53.446382 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:53.446386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:53.446395 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:34:53.446429 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:53.446431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:53.446445 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:53.446578 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:53.446629 (+ 51us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.447871 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.449036 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.449530 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.449591 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.449664 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:53.449725 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 02:34:53.449763 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":57,"threads_started":1}
I20260928 02:34:53.449817 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.445872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39961 (local address 127.0.0.1:41856)
0928 02:34:53.446074 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:53.446085 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:53.446133 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:53.446544 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:53.446545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:53.446547 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:53.446582 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.446583 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.447943 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.447944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.448943 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.448945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.449598 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.449599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.449657 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.449661 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:34:53.449712 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 02:34:53.449752 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182}
I20260928 02:34:53.449913 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.445874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46089 (local address 127.0.0.1:55066)
0928 02:34:53.446137 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:53.446147 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:53.446184 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:53.446406 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:53.446406 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:53.446408 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:53.446443 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.446444 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.447713 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.447714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.448731 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.448733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.449660 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.449661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.449720 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.449723 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:34:53.449749 (+ 26us) client_negotiation.cc:240] Negotiation successful
0928 02:34:53.449826 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":53,"thread_start_us":73,"threads_started":1}
I20260928 02:34:53.450191 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.445975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55066 (local address 127.0.98.2:46089)
0928 02:34:53.446173 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:53.446178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:53.446188 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:34:53.446231 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:53.446233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:53.446247 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:53.446514 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:53.446574 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.447639 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.448811 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.449559 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.449839 (+ 280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.450057 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:53.450082 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 02:34:53.450117 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":92,"threads_started":1}
I20260928 02:34:53.450743 7540 tablet_service.cc:1467] Processing CreateTablet for tablet 5613f27a051743ce954cefec41787777 (DEFAULT_TABLE table=test-table [id=d1b79c2042e34fde96fe428932359093]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:53.451108 7540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5613f27a051743ce954cefec41787777. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:53.451622 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.445972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33037 (local address 127.0.0.1:39910)
0928 02:34:53.447752 (+ 1780us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:53.447761 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:53.447796 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:53.448528 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:53.448529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:53.448530 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:53.448576 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.448577 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.449676 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.449676 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.450676 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.450678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.451478 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.451478 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.451520 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.451524 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:34:53.451554 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 02:34:53.451578 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1766,"thread_start_us":62,"threads_started":1}
I20260928 02:34:53.451673 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.446033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39910 (local address 127.0.98.3:33037)
0928 02:34:53.447886 (+ 1853us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:53.447890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:53.447899 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:34:53.448385 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:53.448388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:53.448404 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:53.448475 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:53.448636 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.449604 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.450759 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.451391 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.451474 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.451539 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:53.451574 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 02:34:53.451607 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1824,"thread_start_us":65,"threads_started":1}
I20260928 02:34:53.452157 7676 tablet_service.cc:1467] Processing CreateTablet for tablet 5613f27a051743ce954cefec41787777 (DEFAULT_TABLE table=test-table [id=d1b79c2042e34fde96fe428932359093]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:53.452406 7676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5613f27a051743ce954cefec41787777. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:53.453685 7403 tablet_service.cc:1467] Processing CreateTablet for tablet 5613f27a051743ce954cefec41787777 (DEFAULT_TABLE table=test-table [id=d1b79c2042e34fde96fe428932359093]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:53.453843 7768 tablet_bootstrap.cc:492] T 5613f27a051743ce954cefec41787777 P f7e8161e260e47cbad68b8e313fe23fd: Bootstrap starting.
I20260928 02:34:53.453969 7403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5613f27a051743ce954cefec41787777. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:53.454298 7768 tablet_bootstrap.cc:654] T 5613f27a051743ce954cefec41787777 P f7e8161e260e47cbad68b8e313fe23fd: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:53.454497 7768 log.cc:824] T 5613f27a051743ce954cefec41787777 P f7e8161e260e47cbad68b8e313fe23fd: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:53.455041 7768 tablet_bootstrap.cc:492] T 5613f27a051743ce954cefec41787777 P f7e8161e260e47cbad68b8e313fe23fd: No bootstrap required, opened a new log
I20260928 02:34:53.455081 7768 ts_tablet_manager.cc:1397] T 5613f27a051743ce954cefec41787777 P f7e8161e260e47cbad68b8e313fe23fd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:34:53.455231 7770 tablet_bootstrap.cc:492] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4: Bootstrap starting.
I20260928 02:34:53.455758 7770 tablet_bootstrap.cc:654] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:53.456048 7770 log.cc:824] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:53.456754 7770 tablet_bootstrap.cc:492] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4: No bootstrap required, opened a new log
I20260928 02:34:53.456807 7770 ts_tablet_manager.cc:1397] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:34:53.456839 7772 tablet_bootstrap.cc:492] T 5613f27a051743ce954cefec41787777 P 2f784af0f7a0463583c819774b174e5c: Bootstrap starting.
I20260928 02:34:53.457839 7772 tablet_bootstrap.cc:654] T 5613f27a051743ce954cefec41787777 P 2f784af0f7a0463583c819774b174e5c: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:53.457818 7768 raft_consensus.cc:348] T 5613f27a051743ce954cefec41787777 P f7e8161e260e47cbad68b8e313fe23fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7e8161e260e47cbad68b8e313fe23fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46089 } } peers { permanent_uuid: "2f784af0f7a0463583c819774b174e5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39961 } } peers { permanent_uuid: "ced49025d91f44928924ee460747c5b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33037 } }
I20260928 02:34:53.457975 7768 raft_consensus.cc:374] T 5613f27a051743ce954cefec41787777 P f7e8161e260e47cbad68b8e313fe23fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:53.458017 7768 raft_consensus.cc:729] T 5613f27a051743ce954cefec41787777 P f7e8161e260e47cbad68b8e313fe23fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7e8161e260e47cbad68b8e313fe23fd, State: Initialized, Role: FOLLOWER
I20260928 02:34:53.458091 7772 log.cc:824] T 5613f27a051743ce954cefec41787777 P 2f784af0f7a0463583c819774b174e5c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:53.458127 7768 consensus_queue.cc:260] T 5613f27a051743ce954cefec41787777 P f7e8161e260e47cbad68b8e313fe23fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7e8161e260e47cbad68b8e313fe23fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46089 } } peers { permanent_uuid: "2f784af0f7a0463583c819774b174e5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39961 } } peers { permanent_uuid: "ced49025d91f44928924ee460747c5b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33037 } }
I20260928 02:34:53.458376 7768 ts_tablet_manager.cc:1428] T 5613f27a051743ce954cefec41787777 P f7e8161e260e47cbad68b8e313fe23fd: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 02:34:53.458492 7606 heartbeater.cc:503] Master 127.0.98.62:34979 was elected leader, sending a full tablet report...
I20260928 02:34:53.458730 7772 tablet_bootstrap.cc:492] T 5613f27a051743ce954cefec41787777 P 2f784af0f7a0463583c819774b174e5c: No bootstrap required, opened a new log
I20260928 02:34:53.458796 7772 ts_tablet_manager.cc:1397] T 5613f27a051743ce954cefec41787777 P 2f784af0f7a0463583c819774b174e5c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 02:34:53.459266 7770 raft_consensus.cc:348] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7e8161e260e47cbad68b8e313fe23fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46089 } } peers { permanent_uuid: "2f784af0f7a0463583c819774b174e5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39961 } } peers { permanent_uuid: "ced49025d91f44928924ee460747c5b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33037 } }
I20260928 02:34:53.459378 7770 raft_consensus.cc:374] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:53.459407 7770 raft_consensus.cc:729] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced49025d91f44928924ee460747c5b4, State: Initialized, Role: FOLLOWER
I20260928 02:34:53.459522 7770 consensus_queue.cc:260] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7e8161e260e47cbad68b8e313fe23fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46089 } } peers { permanent_uuid: "2f784af0f7a0463583c819774b174e5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39961 } } peers { permanent_uuid: "ced49025d91f44928924ee460747c5b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33037 } }
I20260928 02:34:53.459735 7742 heartbeater.cc:503] Master 127.0.98.62:34979 was elected leader, sending a full tablet report...
I20260928 02:34:53.460083 7770 ts_tablet_manager.cc:1428] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:34:53.460484 7772 raft_consensus.cc:348] T 5613f27a051743ce954cefec41787777 P 2f784af0f7a0463583c819774b174e5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7e8161e260e47cbad68b8e313fe23fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46089 } } peers { permanent_uuid: "2f784af0f7a0463583c819774b174e5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39961 } } peers { permanent_uuid: "ced49025d91f44928924ee460747c5b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33037 } }
I20260928 02:34:53.460633 7772 raft_consensus.cc:374] T 5613f27a051743ce954cefec41787777 P 2f784af0f7a0463583c819774b174e5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:53.460675 7772 raft_consensus.cc:729] T 5613f27a051743ce954cefec41787777 P 2f784af0f7a0463583c819774b174e5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f784af0f7a0463583c819774b174e5c, State: Initialized, Role: FOLLOWER
I20260928 02:34:53.460772 7772 consensus_queue.cc:260] T 5613f27a051743ce954cefec41787777 P 2f784af0f7a0463583c819774b174e5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7e8161e260e47cbad68b8e313fe23fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46089 } } peers { permanent_uuid: "2f784af0f7a0463583c819774b174e5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39961 } } peers { permanent_uuid: "ced49025d91f44928924ee460747c5b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33037 } }
I20260928 02:34:53.461001 7772 ts_tablet_manager.cc:1428] T 5613f27a051743ce954cefec41787777 P 2f784af0f7a0463583c819774b174e5c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 02:34:53.461022 7469 heartbeater.cc:503] Master 127.0.98.62:34979 was elected leader, sending a full tablet report...
I20260928 02:34:53.470896 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.468430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60984 (local address 127.0.98.60:41171)
0928 02:34:53.468502 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:53.468504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:53.468513 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:34:53.468535 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:53.468536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:53.468552 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:53.468627 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:53.468766 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.469246 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.470281 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.470668 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.470723 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.470771 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:53.470809 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 02:34:53.470834 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:34:53.470899 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.468339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41171 (local address 127.0.0.1:60984)
0928 02:34:53.468387 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:53.468398 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:53.468437 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:53.468639 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:53.468639 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:53.468641 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:53.468683 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.468685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.469343 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.469344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.470217 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.470220 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.470725 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.470726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.470766 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.470769 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:34:53.470799 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 02:34:53.470833 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 02:34:53.471177 7300 raft_consensus.cc:1203] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260928 02:34:53.471487 7329 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d0b62d9ab8d41ffa729271c52ec57ba. Latest consensus state: current_term: 1 leader_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: false } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: true } } }
I20260928 02:34:53.471593 7329 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:34:53.517571 7775 raft_consensus.cc:482] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:34:53.517692 7775 raft_consensus.cc:504] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7e8161e260e47cbad68b8e313fe23fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46089 } } peers { permanent_uuid: "2f784af0f7a0463583c819774b174e5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39961 } } peers { permanent_uuid: "ced49025d91f44928924ee460747c5b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33037 } }
I20260928 02:34:53.518026 7775 leader_election.cc:288] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7e8161e260e47cbad68b8e313fe23fd (127.0.98.2:46089), 2f784af0f7a0463583c819774b174e5c (127.0.98.1:39961)
I20260928 02:34:53.520654 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.518207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39961 (local address 127.0.98.3:57915)
0928 02:34:53.518279 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:53.518290 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:53.518328 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:53.518471 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:53.518472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:53.518473 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:53.518538 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.518539 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.519115 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.519116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.519952 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.519955 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.520437 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.520437 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.520501 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.520504 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:34:53.520546 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 02:34:53.520583 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":21}
I20260928 02:34:53.520654 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.518330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57915 (local address 127.0.98.1:39961)
0928 02:34:53.518369 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:53.518372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:53.518379 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:34:53.518404 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:53.518404 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:53.518415 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:53.518457 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:53.518593 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.519048 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.520027 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.520370 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.520429 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.520439 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:53.520558 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 02:34:53.520593 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:34:53.520790 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.518314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43345 (local address 127.0.98.2:46089)
0928 02:34:53.518356 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:53.518358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:53.518477 (+ 119us) server_negotiation.cc:407] Connection header received
0928 02:34:53.518532 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:53.518533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:53.518544 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:53.518596 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:53.518718 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.519171 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.520141 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.520495 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.520553 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.520562 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:53.520698 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 02:34:53.520730 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:34:53.520800 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.518226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46089 (local address 127.0.98.3:43345)
0928 02:34:53.518417 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:53.518428 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:53.518482 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:53.518601 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:53.518601 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:53.518605 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:53.518665 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.518666 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.519237 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.519238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.520067 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:53.520069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.520566 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.520567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:53.520644 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.520646 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:34:53.520689 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 02:34:53.520723 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":146}
I20260928 02:34:53.521046 7560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5613f27a051743ce954cefec41787777" candidate_uuid: "ced49025d91f44928924ee460747c5b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7e8161e260e47cbad68b8e313fe23fd" is_pre_election: true
I20260928 02:34:53.521193 7560 raft_consensus.cc:2454] T 5613f27a051743ce954cefec41787777 P f7e8161e260e47cbad68b8e313fe23fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced49025d91f44928924ee460747c5b4 in term 0.
I20260928 02:34:53.521200 7423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5613f27a051743ce954cefec41787777" candidate_uuid: "ced49025d91f44928924ee460747c5b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f784af0f7a0463583c819774b174e5c" is_pre_election: true
I20260928 02:34:53.521358 7423 raft_consensus.cc:2454] T 5613f27a051743ce954cefec41787777 P 2f784af0f7a0463583c819774b174e5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced49025d91f44928924ee460747c5b4 in term 0.
I20260928 02:34:53.521417 7631 leader_election.cc:302] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ced49025d91f44928924ee460747c5b4, f7e8161e260e47cbad68b8e313fe23fd; no voters:
I20260928 02:34:53.521543 7775 raft_consensus.cc:2781] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:34:53.521613 7775 raft_consensus.cc:482] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:34:53.521631 7775 raft_consensus.cc:3037] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:53.522255 7775 raft_consensus.cc:504] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7e8161e260e47cbad68b8e313fe23fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46089 } } peers { permanent_uuid: "2f784af0f7a0463583c819774b174e5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39961 } } peers { permanent_uuid: "ced49025d91f44928924ee460747c5b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33037 } }
I20260928 02:34:53.522364 7775 leader_election.cc:288] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4 [CANDIDATE]: Term 1 election: Requested vote from peers f7e8161e260e47cbad68b8e313fe23fd (127.0.98.2:46089), 2f784af0f7a0463583c819774b174e5c (127.0.98.1:39961)
I20260928 02:34:53.522500 7423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5613f27a051743ce954cefec41787777" candidate_uuid: "ced49025d91f44928924ee460747c5b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f784af0f7a0463583c819774b174e5c"
I20260928 02:34:53.522495 7560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5613f27a051743ce954cefec41787777" candidate_uuid: "ced49025d91f44928924ee460747c5b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7e8161e260e47cbad68b8e313fe23fd"
I20260928 02:34:53.522571 7423 raft_consensus.cc:3037] T 5613f27a051743ce954cefec41787777 P 2f784af0f7a0463583c819774b174e5c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:53.522571 7560 raft_consensus.cc:3037] T 5613f27a051743ce954cefec41787777 P f7e8161e260e47cbad68b8e313fe23fd [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:53.523334 7423 raft_consensus.cc:2454] T 5613f27a051743ce954cefec41787777 P 2f784af0f7a0463583c819774b174e5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced49025d91f44928924ee460747c5b4 in term 1.
I20260928 02:34:53.523334 7560 raft_consensus.cc:2454] T 5613f27a051743ce954cefec41787777 P f7e8161e260e47cbad68b8e313fe23fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced49025d91f44928924ee460747c5b4 in term 1.
I20260928 02:34:53.523504 7631 leader_election.cc:302] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ced49025d91f44928924ee460747c5b4, f7e8161e260e47cbad68b8e313fe23fd; no voters:
I20260928 02:34:53.523597 7775 raft_consensus.cc:2781] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:53.523759 7775 raft_consensus.cc:686] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4 [term 1 LEADER]: Becoming Leader. State: Replica: ced49025d91f44928924ee460747c5b4, State: Running, Role: LEADER
I20260928 02:34:53.523866 7775 consensus_queue.cc:237] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7e8161e260e47cbad68b8e313fe23fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46089 } } peers { permanent_uuid: "2f784af0f7a0463583c819774b174e5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39961 } } peers { permanent_uuid: "ced49025d91f44928924ee460747c5b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33037 } }
I20260928 02:34:53.524621 6499 catalog_manager.cc:5495] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ced49025d91f44928924ee460747c5b4 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ced49025d91f44928924ee460747c5b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7e8161e260e47cbad68b8e313fe23fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f784af0f7a0463583c819774b174e5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ced49025d91f44928924ee460747c5b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33037 } health_report { overall_health: HEALTHY } } }
I20260928 02:34:53.525131 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: false } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: true } } }
I20260928 02:34:53.525223 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:53.525807 7058 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba: attempting to promote NON_VOTER c6b3724cda7546ca9a2c651f5c1a0688 to VOTER
I20260928 02:34:53.525938 7058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: false } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: false } }
I20260928 02:34:53.526324 7108 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [term 1 FOLLOWER]: Refusing update from remote peer 6d0b62d9ab8d41ffa729271c52ec57ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 02:34:53.526521 7059 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260928 02:34:53.526561 7300 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [term 1 LEARNER]: Refusing update from remote peer 6d0b62d9ab8d41ffa729271c52ec57ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260928 02:34:53.526721 7059 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260928 02:34:53.527124 7783 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, c6b3724cda7546ca9a2c651f5c1a0688 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: false } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: false } } }
I20260928 02:34:53.527228 7108 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, c6b3724cda7546ca9a2c651f5c1a0688 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: false } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: false } } }
I20260928 02:34:53.527335 7300 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, c6b3724cda7546ca9a2c651f5c1a0688 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: false } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: false } } }
I20260928 02:34:53.528069 7254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: false } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: false } } }
I20260928 02:34:53.528064 7783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: false } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: false } } }
I20260928 02:34:53.528131 7254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:34:53.528131 7783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:53.528088 7060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: false } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: false } } }
I20260928 02:34:53.528167 7060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:53.528151 7329 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: false } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: false } } }
I20260928 02:34:53.528216 7329 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:34:53.528194 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: false } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: false } } }
I20260928 02:34:53.528239 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:53.543848 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.541613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39916 (local address 127.0.98.3:33037)
0928 02:34:53.541657 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:53.541659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:53.541669 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:34:53.541705 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:53.541705 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:53.541726 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:53.541806 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:34:53.541904 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.542369 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.543091 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.543297 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.543350 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.543491 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:34:53.543634 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:34:53.543703 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:53.543759 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 02:34:53.543777 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:34:53.545513 7423 raft_consensus.cc:1261] T 5613f27a051743ce954cefec41787777 P 2f784af0f7a0463583c819774b174e5c [term 1 FOLLOWER]: Refusing update from remote peer ced49025d91f44928924ee460747c5b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:53.545533 7560 raft_consensus.cc:1261] T 5613f27a051743ce954cefec41787777 P f7e8161e260e47cbad68b8e313fe23fd [term 1 FOLLOWER]: Refusing update from remote peer ced49025d91f44928924ee460747c5b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:53.545825 7775 consensus_queue.cc:1035] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f784af0f7a0463583c819774b174e5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:53.545959 7781 consensus_queue.cc:1035] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7e8161e260e47cbad68b8e313fe23fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:53.548439 7785 mvcc.cc:205] Tried to move back new op lower bound from 7334145611959848960 to 7334145611874476032. Current Snapshot: MvccSnapshot[applied={T|T < 7334145611959848960}]
I20260928 02:34:53.549362 7787 mvcc.cc:205] Tried to move back new op lower bound from 7334145611959848960 to 7334145611874476032. Current Snapshot: MvccSnapshot[applied={T|T < 7334145611959848960}]
I20260928 02:34:53.551514 6499 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:34:53.551608 6499 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:34:53.552820 6499 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d0b62d9ab8d41ffa729271c52ec57ba: Sending DeleteTablet for 3 replicas of tablet 5613f27a051743ce954cefec41787777
I20260928 02:34:53.553453 7403 tablet_service.cc:1514] Processing DeleteTablet for tablet 5613f27a051743ce954cefec41787777 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41856
I20260928 02:34:53.553659 7676 tablet_service.cc:1514] Processing DeleteTablet for tablet 5613f27a051743ce954cefec41787777 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39910
I20260928 02:34:53.553851 7797 tablet_replica.cc:330] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4: stopping tablet replica
I20260928 02:34:53.553941 7797 raft_consensus.cc:2229] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:34:53.554023 7797 raft_consensus.cc:2258] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:53.554225 7540 tablet_service.cc:1514] Processing DeleteTablet for tablet 5613f27a051743ce954cefec41787777 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:53 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55066
I20260928 02:34:53.554601 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 6468
I20260928 02:34:53.554658 7797 ts_tablet_manager.cc:1905] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:53.556185 7797 ts_tablet_manager.cc:1918] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:34:53.556247 7797 log.cc:1196] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/wal/wals/5613f27a051743ce954cefec41787777
I20260928 02:34:53.556471 7797 ts_tablet_manager.cc:1939] T 5613f27a051743ce954cefec41787777 P ced49025d91f44928924ee460747c5b4: Deleting consensus metadata
I20260928 02:34:53.556874 7798 tablet_replica.cc:330] T 5613f27a051743ce954cefec41787777 P f7e8161e260e47cbad68b8e313fe23fd: stopping tablet replica
I20260928 02:34:53.556962 7798 raft_consensus.cc:2229] T 5613f27a051743ce954cefec41787777 P f7e8161e260e47cbad68b8e313fe23fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:34:53.557057 7798 raft_consensus.cc:2258] T 5613f27a051743ce954cefec41787777 P f7e8161e260e47cbad68b8e313fe23fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:53.557464 7798 ts_tablet_manager.cc:1905] T 5613f27a051743ce954cefec41787777 P f7e8161e260e47cbad68b8e313fe23fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:53.558818 7798 ts_tablet_manager.cc:1918] T 5613f27a051743ce954cefec41787777 P f7e8161e260e47cbad68b8e313fe23fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:34:53.558873 7798 log.cc:1196] T 5613f27a051743ce954cefec41787777 P f7e8161e260e47cbad68b8e313fe23fd: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/wal/wals/5613f27a051743ce954cefec41787777
I20260928 02:34:53.559098 7798 ts_tablet_manager.cc:1939] T 5613f27a051743ce954cefec41787777 P f7e8161e260e47cbad68b8e313fe23fd: Deleting consensus metadata
W20260928 02:34:53.561028 7629 connection.cc:535] server connection from 127.0.0.1:39910 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 02:34:53.561406 7495 connection.cc:535] server connection from 127.0.0.1:55066 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:34:53.561641 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.553857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60998 (local address 127.0.98.60:41171)
0928 02:34:53.553896 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:53.553898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:53.553905 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:34:53.553924 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:53.553924 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:53.553933 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:53.553985 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:53.555094 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.555825 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.556884 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.557185 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.557226 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.558783 (+ 1557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:53.558784 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:53.558785 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:53.558791 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:53.560388 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.560758 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.560759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.560759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.560820 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.561035 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.561036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.561037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.561166 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:53.561278 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:53.561467 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 02:34:53.561546 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:34:53.562052 7796 tablet_replica.cc:330] T 5613f27a051743ce954cefec41787777 P 2f784af0f7a0463583c819774b174e5c: stopping tablet replica
I20260928 02:34:53.562155 7796 raft_consensus.cc:2229] T 5613f27a051743ce954cefec41787777 P 2f784af0f7a0463583c819774b174e5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:34:53.562258 7796 raft_consensus.cc:2258] T 5613f27a051743ce954cefec41787777 P 2f784af0f7a0463583c819774b174e5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:53.563374 7796 ts_tablet_manager.cc:1905] T 5613f27a051743ce954cefec41787777 P 2f784af0f7a0463583c819774b174e5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:53.564849 7796 ts_tablet_manager.cc:1918] T 5613f27a051743ce954cefec41787777 P 2f784af0f7a0463583c819774b174e5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:34:53.564920 7796 log.cc:1196] T 5613f27a051743ce954cefec41787777 P 2f784af0f7a0463583c819774b174e5c: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-0/wal/wals/5613f27a051743ce954cefec41787777
I20260928 02:34:53.565164 7796 ts_tablet_manager.cc:1939] T 5613f27a051743ce954cefec41787777 P 2f784af0f7a0463583c819774b174e5c: Deleting consensus metadata
W20260928 02:34:53.565536 7358 connection.cc:441] server connection from 127.0.0.1:41856 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41856 (request call id 1) could send its response
I20260928 02:34:53.569921 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.562853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46980 (local address 127.0.98.61:45279)
0928 02:34:53.562893 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:53.562895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:53.563112 (+ 217us) server_negotiation.cc:407] Connection header received
0928 02:34:53.563186 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:53.563187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:53.563201 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:53.563246 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:53.563769 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.564252 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.564983 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.565233 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.565306 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.566661 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:53.566662 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:53.566663 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:53.566670 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:53.568660 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.569000 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.569001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.569001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.569063 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.569288 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.569290 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.569290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.569406 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:53.569516 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:53.569695 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 02:34:53.569853 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:34:53.571518 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:53.563062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32780 (local address 127.0.98.60:41171)
0928 02:34:53.565373 (+ 2311us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:53.565375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:53.565381 (+ 6us) server_negotiation.cc:407] Connection header received
0928 02:34:53.565397 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:53.565397 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:53.565405 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:53.565449 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:53.566364 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.566919 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.567677 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:53.567900 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:53.567952 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:53.569100 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:53.569101 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:53.569101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:53.569107 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:53.570611 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.570892 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.570893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.570894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.570945 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:53.571104 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:53.571105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:53.571105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:53.571202 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:53.571277 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:53.571377 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 02:34:53.571435 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2288}
I20260928 02:34:54.549882 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:54.549553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.98.3:40697)
0928 02:34:54.549690 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:54.549784 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
W20260928 02:34:54.550099 7742 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34979 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111)
I20260928 02:34:54.550125 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:54.549897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.98.2:54291)
0928 02:34:54.550020 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:54.550083 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":50,"threads_started":1}
W20260928 02:34:54.550302 7606 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34979 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111)
I20260928 02:34:54.551011 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:54.550800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.98.1:38651)
0928 02:34:54.550910 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:54.550968 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
W20260928 02:34:54.551175 7469 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34979 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111)
I20260928 02:34:55.053014 7809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d0b62d9ab8d41ffa729271c52ec57ba)
I20260928 02:34:55.053165 7809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: false } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: false } }
I20260928 02:34:55.053442 7809 leader_election.cc:288] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d0b62d9ab8d41ffa729271c52ec57ba (127.0.98.62:34979), c6b3724cda7546ca9a2c651f5c1a0688 (127.0.98.60:41171)
I20260928 02:34:55.053545 7810 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6d0b62d9ab8d41ffa729271c52ec57ba)
I20260928 02:34:55.053670 7810 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: false } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: false } }
I20260928 02:34:55.053797 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.053550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.0.1:35546)
0928 02:34:55.053712 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:55.053758 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":64,"threads_started":1}
W20260928 02:34:55.053902 7084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111)
I20260928 02:34:55.054065 7810 leader_election.cc:288] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6d0b62d9ab8d41ffa729271c52ec57ba (127.0.98.62:34979), 37118a91babb433594ad029895cee4a7 (127.0.98.61:45279)
I20260928 02:34:55.054235 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.053960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.0.1:35556)
0928 02:34:55.054157 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:55.054202 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":50,"threads_started":1}
I20260928 02:34:55.054253 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.054183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.0.1:35560)
0928 02:34:55.054217 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:55.054234 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 02:34:55.054313 7084 leader_election.cc:334] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d0b62d9ab8d41ffa729271c52ec57ba (127.0.98.62:34979): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111)
W20260928 02:34:55.054319 7276 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111)
I20260928 02:34:55.054695 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.054621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.0.1:35562)
0928 02:34:55.054653 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:55.054672 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260928 02:34:55.054764 7276 leader_election.cc:334] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6d0b62d9ab8d41ffa729271c52ec57ba (127.0.98.62:34979): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111)
I20260928 02:34:55.056749 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.053743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32794 (local address 127.0.98.60:41171)
0928 02:34:55.053828 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:55.053896 (+ 68us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:55.053905 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:34:55.053999 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:55.054000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:55.054010 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:55.054059 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:55.054233 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:55.054934 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:55.056000 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:55.056496 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:55.056544 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:55.056553 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:55.056673 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 02:34:55.056700 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65,"thread_start_us":45,"threads_started":1}
I20260928 02:34:55.056774 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.053599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41171 (local address 127.0.0.1:32794)
0928 02:34:55.053839 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:55.053849 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:55.053906 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:55.054108 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:55.054109 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:55.054111 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:55.054164 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:55.054166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:55.055004 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:55.055005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:55.055929 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:55.055931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:55.056553 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:55.056554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:55.056613 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:55.056617 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:34:55.056666 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 02:34:55.056708 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":49,"threads_started":1}
I20260928 02:34:55.057039 7300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "37118a91babb433594ad029895cee4a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" is_pre_election: true
I20260928 02:34:55.057137 7300 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37118a91babb433594ad029895cee4a7 in term 1.
I20260928 02:34:55.057228 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.054329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46990 (local address 127.0.98.61:45279)
0928 02:34:55.054543 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:55.054544 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:55.054552 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:34:55.054571 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:55.054572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:55.054581 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:55.054649 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:55.054791 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:55.055450 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:55.056455 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:55.056933 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:55.056993 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:55.057003 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:55.057137 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 02:34:55.057174 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":73,"threads_started":1}
I20260928 02:34:55.057231 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.054154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45279 (local address 127.0.0.1:46990)
0928 02:34:55.054406 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:55.054417 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:55.054468 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:55.054662 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:55.054663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:55.054665 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:55.054709 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:55.054711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:55.055517 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:55.055518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:55.056382 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:55.056384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:55.057003 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:55.057004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:55.057068 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:55.057073 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:34:55.057122 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 02:34:55.057159 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":45,"threads_started":1}
I20260928 02:34:55.057312 7083 leader_election.cc:302] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37118a91babb433594ad029895cee4a7, c6b3724cda7546ca9a2c651f5c1a0688; no voters: 6d0b62d9ab8d41ffa729271c52ec57ba
I20260928 02:34:55.057413 7809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 02:34:55.057456 7809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6d0b62d9ab8d41ffa729271c52ec57ba)
I20260928 02:34:55.057473 7809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:34:55.057502 7108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "37118a91babb433594ad029895cee4a7" is_pre_election: true
I20260928 02:34:55.058174 7809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: false } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: false } }
I20260928 02:34:55.058287 7108 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c6b3724cda7546ca9a2c651f5c1a0688 in current term 2: Already voted for candidate 37118a91babb433594ad029895cee4a7 in this term.
I20260928 02:34:55.058290 7809 leader_election.cc:288] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [CANDIDATE]: Term 2 election: Requested vote from peers 6d0b62d9ab8d41ffa729271c52ec57ba (127.0.98.62:34979), c6b3724cda7546ca9a2c651f5c1a0688 (127.0.98.60:41171)
I20260928 02:34:55.058454 7300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "37118a91babb433594ad029895cee4a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c6b3724cda7546ca9a2c651f5c1a0688"
I20260928 02:34:55.058497 7277 leader_election.cc:302] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6b3724cda7546ca9a2c651f5c1a0688; no voters: 37118a91babb433594ad029895cee4a7, 6d0b62d9ab8d41ffa729271c52ec57ba
I20260928 02:34:55.058518 7300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:34:55.058580 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.058358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.0.1:35568)
0928 02:34:55.058414 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:55.058559 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 02:34:55.058903 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.058820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.0.1:35570)
0928 02:34:55.058859 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:55.058885 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 02:34:55.058979 7084 leader_election.cc:334] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6d0b62d9ab8d41ffa729271c52ec57ba (127.0.98.62:34979): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111)
I20260928 02:34:55.059263 7300 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37118a91babb433594ad029895cee4a7 in term 2.
I20260928 02:34:55.059333 7810 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:34:55.059402 7083 leader_election.cc:302] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37118a91babb433594ad029895cee4a7, c6b3724cda7546ca9a2c651f5c1a0688; no voters: 6d0b62d9ab8d41ffa729271c52ec57ba
I20260928 02:34:55.059489 7809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:34:55.059593 7809 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [term 2 LEADER]: Becoming Leader. State: Replica: 37118a91babb433594ad029895cee4a7, State: Running, Role: LEADER
I20260928 02:34:55.059659 7809 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: false } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: false } }
I20260928 02:34:55.059947 7819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37118a91babb433594ad029895cee4a7. Latest consensus state: current_term: 2 leader_uuid: "37118a91babb433594ad029895cee4a7" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: false } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: false } } }
I20260928 02:34:55.060030 7819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:55.060170 7821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:34:55.060537 7821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:34:55.060591 7821 catalog_manager.cc:1259] Loaded cluster ID: ef1a356553cc4e3597de38b414bb0511
I20260928 02:34:55.060611 7821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:34:55.060721 7821 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:34:55.060771 7821 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7: Loaded TSK: 0
I20260928 02:34:55.060858 7821 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:34:55.099509 7097 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:34:55.101070 7097 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:34:55.101997 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.101890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.0.1:35584)
0928 02:34:55.101935 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:55.101967 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 02:34:55.102032 7300 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [term 2 FOLLOWER]: Refusing update from remote peer 37118a91babb433594ad029895cee4a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 02:34:55.102317 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.102212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.0.1:35600)
0928 02:34:55.102240 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:55.102263 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
I20260928 02:34:55.102272 7819 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260928 02:34:55.102442 7084 consensus_peers.cc:487] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 -> Peer 6d0b62d9ab8d41ffa729271c52ec57ba (127.0.98.62:34979): Couldn't send request to peer 6d0b62d9ab8d41ffa729271c52ec57ba. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 02:34:55.102834 7810 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37118a91babb433594ad029895cee4a7. Latest consensus state: current_term: 2 leader_uuid: "37118a91babb433594ad029895cee4a7" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: false } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: false } } }
I20260928 02:34:55.102937 7810 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:34:55.103361 7822 mvcc.cc:205] Tried to move back new op lower bound from 7334145618336227328 to 7334145618164985856. Current Snapshot: MvccSnapshot[applied={T|T < 7334145618336227328}]
I20260928 02:34:55.103423 7809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "37118a91babb433594ad029895cee4a7" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: false } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: false } } }
I20260928 02:34:55.103508 7810 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "37118a91babb433594ad029895cee4a7" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6d0b62d9ab8d41ffa729271c52ec57ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } peers { permanent_uuid: "37118a91babb433594ad029895cee4a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45279 } attrs { promote: false } } peers { permanent_uuid: "c6b3724cda7546ca9a2c651f5c1a0688" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41171 } attrs { promote: false } } }
I20260928 02:34:55.103559 7809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7 [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:55.103579 7810 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b3724cda7546ca9a2c651f5c1a0688 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:34:55.108098 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.105309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39928 (local address 127.0.98.3:33037)
0928 02:34:55.105403 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:55.105404 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:55.105569 (+ 165us) server_negotiation.cc:407] Connection header received
0928 02:34:55.105626 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:55.105626 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:55.105637 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:55.105689 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:55.105840 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:55.106505 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:55.107343 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:55.107822 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:55.107880 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:55.107891 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:55.108002 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 02:34:55.108033 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":54,"threads_started":1}
I20260928 02:34:55.108096 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.105157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33037 (local address 127.0.0.1:39928)
0928 02:34:55.105520 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:55.105529 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:55.105579 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:55.105734 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:55.105735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:55.105737 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:55.105770 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:55.105771 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:55.106569 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:55.106570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:55.107269 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:55.107270 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:55.107886 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:55.107887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:55.107946 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:55.107950 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:34:55.107993 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 02:34:55.108030 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":62,"threads_started":1}
I20260928 02:34:55.108384 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.105087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46089 (local address 127.0.0.1:55070)
0928 02:34:55.105248 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:55.105274 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:55.105304 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:55.105432 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:55.105433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:55.105435 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:55.105481 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:55.105482 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:55.106280 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:55.106281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:55.107427 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:55.107429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:55.108184 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:55.108185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:55.108249 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:55.108253 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:34:55.108280 (+ 27us) client_negotiation.cc:240] Negotiation successful
0928 02:34:55.108313 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":28}
I20260928 02:34:55.108402 7676 tablet_service.cc:1467] Processing CreateTablet for tablet e2a4ba5587bd418899e2cf66f8cf6b2a (DEFAULT_TABLE table=test-table [id=307c41b28a8f4d8f9814ec3fdbfd9edc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:55.108480 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.105230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41858 (local address 127.0.98.1:39961)
0928 02:34:55.105352 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:55.105354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:55.105361 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:34:55.105384 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:55.105385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:55.105394 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:55.105456 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:55.105576 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:55.106326 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:55.107537 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:55.108095 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:55.108352 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:55.108365 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:55.108382 (+ 17us) server_negotiation.cc:299] Negotiation successful
0928 02:34:55.108416 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
I20260928 02:34:55.108507 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.105058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39961 (local address 127.0.0.1:41858)
0928 02:34:55.105125 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:55.105134 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:55.105179 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:55.105465 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:55.105465 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:55.105468 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:55.105501 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:55.105502 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:55.106394 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:55.106394 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:55.107446 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:55.107447 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:55.108184 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:55.108185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:55.108249 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:55.108252 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:34:55.108280 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 02:34:55.108313 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":1}
I20260928 02:34:55.108530 7676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2a4ba5587bd418899e2cf66f8cf6b2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:55.108737 7403 tablet_service.cc:1467] Processing CreateTablet for tablet e2a4ba5587bd418899e2cf66f8cf6b2a (DEFAULT_TABLE table=test-table [id=307c41b28a8f4d8f9814ec3fdbfd9edc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:55.108800 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.105185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55070 (local address 127.0.98.2:46089)
0928 02:34:55.105311 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:55.105312 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:55.105321 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:34:55.105349 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:55.105350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:55.105360 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:55.105432 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:55.105545 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:55.106206 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:55.107539 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:55.108096 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:55.108674 (+ 578us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:55.108688 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:55.108704 (+ 16us) server_negotiation.cc:299] Negotiation successful
0928 02:34:55.108738 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":54,"threads_started":1}
I20260928 02:34:55.108847 7403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2a4ba5587bd418899e2cf66f8cf6b2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:55.109803 7831 tablet_bootstrap.cc:492] T e2a4ba5587bd418899e2cf66f8cf6b2a P ced49025d91f44928924ee460747c5b4: Bootstrap starting.
I20260928 02:34:55.109905 7832 tablet_bootstrap.cc:492] T e2a4ba5587bd418899e2cf66f8cf6b2a P 2f784af0f7a0463583c819774b174e5c: Bootstrap starting.
I20260928 02:34:55.110001 7540 tablet_service.cc:1467] Processing CreateTablet for tablet e2a4ba5587bd418899e2cf66f8cf6b2a (DEFAULT_TABLE table=test-table [id=307c41b28a8f4d8f9814ec3fdbfd9edc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:55.110111 7540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2a4ba5587bd418899e2cf66f8cf6b2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:55.110277 7832 tablet_bootstrap.cc:654] T e2a4ba5587bd418899e2cf66f8cf6b2a P 2f784af0f7a0463583c819774b174e5c: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:55.110296 7831 tablet_bootstrap.cc:654] T e2a4ba5587bd418899e2cf66f8cf6b2a P ced49025d91f44928924ee460747c5b4: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:55.110739 7831 tablet_bootstrap.cc:492] T e2a4ba5587bd418899e2cf66f8cf6b2a P ced49025d91f44928924ee460747c5b4: No bootstrap required, opened a new log
I20260928 02:34:55.110769 7832 tablet_bootstrap.cc:492] T e2a4ba5587bd418899e2cf66f8cf6b2a P 2f784af0f7a0463583c819774b174e5c: No bootstrap required, opened a new log
I20260928 02:34:55.110798 7831 ts_tablet_manager.cc:1397] T e2a4ba5587bd418899e2cf66f8cf6b2a P ced49025d91f44928924ee460747c5b4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:34:55.110826 7832 ts_tablet_manager.cc:1397] T e2a4ba5587bd418899e2cf66f8cf6b2a P 2f784af0f7a0463583c819774b174e5c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:34:55.110906 7831 raft_consensus.cc:348] T e2a4ba5587bd418899e2cf66f8cf6b2a P ced49025d91f44928924ee460747c5b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f784af0f7a0463583c819774b174e5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39961 } } peers { permanent_uuid: "ced49025d91f44928924ee460747c5b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33037 } } peers { permanent_uuid: "f7e8161e260e47cbad68b8e313fe23fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46089 } }
I20260928 02:34:55.110975 7831 raft_consensus.cc:374] T e2a4ba5587bd418899e2cf66f8cf6b2a P ced49025d91f44928924ee460747c5b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:55.110945 7832 raft_consensus.cc:348] T e2a4ba5587bd418899e2cf66f8cf6b2a P 2f784af0f7a0463583c819774b174e5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f784af0f7a0463583c819774b174e5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39961 } } peers { permanent_uuid: "ced49025d91f44928924ee460747c5b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33037 } } peers { permanent_uuid: "f7e8161e260e47cbad68b8e313fe23fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46089 } }
I20260928 02:34:55.110994 7831 raft_consensus.cc:729] T e2a4ba5587bd418899e2cf66f8cf6b2a P ced49025d91f44928924ee460747c5b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced49025d91f44928924ee460747c5b4, State: Initialized, Role: FOLLOWER
I20260928 02:34:55.111001 7832 raft_consensus.cc:374] T e2a4ba5587bd418899e2cf66f8cf6b2a P 2f784af0f7a0463583c819774b174e5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:55.111018 7832 raft_consensus.cc:729] T e2a4ba5587bd418899e2cf66f8cf6b2a P 2f784af0f7a0463583c819774b174e5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f784af0f7a0463583c819774b174e5c, State: Initialized, Role: FOLLOWER
I20260928 02:34:55.111056 7832 consensus_queue.cc:260] T e2a4ba5587bd418899e2cf66f8cf6b2a P 2f784af0f7a0463583c819774b174e5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f784af0f7a0463583c819774b174e5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39961 } } peers { permanent_uuid: "ced49025d91f44928924ee460747c5b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33037 } } peers { permanent_uuid: "f7e8161e260e47cbad68b8e313fe23fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46089 } }
I20260928 02:34:55.111080 7831 consensus_queue.cc:260] T e2a4ba5587bd418899e2cf66f8cf6b2a P ced49025d91f44928924ee460747c5b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f784af0f7a0463583c819774b174e5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39961 } } peers { permanent_uuid: "ced49025d91f44928924ee460747c5b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33037 } } peers { permanent_uuid: "f7e8161e260e47cbad68b8e313fe23fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46089 } }
I20260928 02:34:55.111184 7832 ts_tablet_manager.cc:1428] T e2a4ba5587bd418899e2cf66f8cf6b2a P 2f784af0f7a0463583c819774b174e5c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:55.111209 7831 ts_tablet_manager.cc:1428] T e2a4ba5587bd418899e2cf66f8cf6b2a P ced49025d91f44928924ee460747c5b4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:55.111662 7837 tablet_bootstrap.cc:492] T e2a4ba5587bd418899e2cf66f8cf6b2a P f7e8161e260e47cbad68b8e313fe23fd: Bootstrap starting.
I20260928 02:34:55.111758 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.111376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.98.1:56371)
0928 02:34:55.111684 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:55.111717 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":78,"threads_started":1}
I20260928 02:34:55.112013 7837 tablet_bootstrap.cc:654] T e2a4ba5587bd418899e2cf66f8cf6b2a P f7e8161e260e47cbad68b8e313fe23fd: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:55.112030 7744 heartbeater.cc:503] Master 127.0.98.61:45279 was elected leader, sending a full tablet report...
I20260928 02:34:55.112317 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.111553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.98.3:56821)
0928 02:34:55.112252 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:55.112285 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":68,"threads_started":1}
I20260928 02:34:55.112475 7471 heartbeater.cc:503] Master 127.0.98.61:45279 was elected leader, sending a full tablet report...
I20260928 02:34:55.112617 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.112527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.98.1:39755)
0928 02:34:55.112573 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:55.112595 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 02:34:55.112576 7837 tablet_bootstrap.cc:492] T e2a4ba5587bd418899e2cf66f8cf6b2a P f7e8161e260e47cbad68b8e313fe23fd: No bootstrap required, opened a new log
I20260928 02:34:55.112758 7837 ts_tablet_manager.cc:1397] T e2a4ba5587bd418899e2cf66f8cf6b2a P f7e8161e260e47cbad68b8e313fe23fd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:34:55.112870 7837 raft_consensus.cc:348] T e2a4ba5587bd418899e2cf66f8cf6b2a P f7e8161e260e47cbad68b8e313fe23fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f784af0f7a0463583c819774b174e5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39961 } } peers { permanent_uuid: "ced49025d91f44928924ee460747c5b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33037 } } peers { permanent_uuid: "f7e8161e260e47cbad68b8e313fe23fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46089 } }
I20260928 02:34:55.112948 7837 raft_consensus.cc:374] T e2a4ba5587bd418899e2cf66f8cf6b2a P f7e8161e260e47cbad68b8e313fe23fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:55.112972 7837 raft_consensus.cc:729] T e2a4ba5587bd418899e2cf66f8cf6b2a P f7e8161e260e47cbad68b8e313fe23fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7e8161e260e47cbad68b8e313fe23fd, State: Initialized, Role: FOLLOWER
I20260928 02:34:55.113023 7837 consensus_queue.cc:260] T e2a4ba5587bd418899e2cf66f8cf6b2a P f7e8161e260e47cbad68b8e313fe23fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f784af0f7a0463583c819774b174e5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39961 } } peers { permanent_uuid: "ced49025d91f44928924ee460747c5b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33037 } } peers { permanent_uuid: "f7e8161e260e47cbad68b8e313fe23fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46089 } }
I20260928 02:34:55.113111 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.112562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.98.3:55383)
0928 02:34:55.112909 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:55.112938 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332}
I20260928 02:34:55.113157 7837 ts_tablet_manager.cc:1428] T e2a4ba5587bd418899e2cf66f8cf6b2a P f7e8161e260e47cbad68b8e313fe23fd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:55.113590 7608 heartbeater.cc:503] Master 127.0.98.61:45279 was elected leader, sending a full tablet report...
I20260928 02:34:55.113610 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.113288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.98.2:46485)
0928 02:34:55.113390 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:55.113418 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":68,"threads_started":1}
I20260928 02:34:55.113884 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.113805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.98.2:40477)
0928 02:34:55.113835 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:55.113862 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260928 02:34:55.129618 7836 raft_consensus.cc:482] T e2a4ba5587bd418899e2cf66f8cf6b2a P ced49025d91f44928924ee460747c5b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:34:55.129680 7836 raft_consensus.cc:504] T e2a4ba5587bd418899e2cf66f8cf6b2a P ced49025d91f44928924ee460747c5b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f784af0f7a0463583c819774b174e5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39961 } } peers { permanent_uuid: "ced49025d91f44928924ee460747c5b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33037 } } peers { permanent_uuid: "f7e8161e260e47cbad68b8e313fe23fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46089 } }
I20260928 02:34:55.129803 7836 leader_election.cc:288] T e2a4ba5587bd418899e2cf66f8cf6b2a P ced49025d91f44928924ee460747c5b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f784af0f7a0463583c819774b174e5c (127.0.98.1:39961), f7e8161e260e47cbad68b8e313fe23fd (127.0.98.2:46089)
I20260928 02:34:55.129966 7423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2a4ba5587bd418899e2cf66f8cf6b2a" candidate_uuid: "ced49025d91f44928924ee460747c5b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f784af0f7a0463583c819774b174e5c" is_pre_election: true
I20260928 02:34:55.129992 7560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2a4ba5587bd418899e2cf66f8cf6b2a" candidate_uuid: "ced49025d91f44928924ee460747c5b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7e8161e260e47cbad68b8e313fe23fd" is_pre_election: true
I20260928 02:34:55.130059 7423 raft_consensus.cc:2454] T e2a4ba5587bd418899e2cf66f8cf6b2a P 2f784af0f7a0463583c819774b174e5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced49025d91f44928924ee460747c5b4 in term 0.
I20260928 02:34:55.130059 7560 raft_consensus.cc:2454] T e2a4ba5587bd418899e2cf66f8cf6b2a P f7e8161e260e47cbad68b8e313fe23fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ced49025d91f44928924ee460747c5b4 in term 0.
I20260928 02:34:55.130188 7629 leader_election.cc:302] T e2a4ba5587bd418899e2cf66f8cf6b2a P ced49025d91f44928924ee460747c5b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f784af0f7a0463583c819774b174e5c, ced49025d91f44928924ee460747c5b4; no voters:
I20260928 02:34:55.130282 7836 raft_consensus.cc:2781] T e2a4ba5587bd418899e2cf66f8cf6b2a P ced49025d91f44928924ee460747c5b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:34:55.130321 7836 raft_consensus.cc:482] T e2a4ba5587bd418899e2cf66f8cf6b2a P ced49025d91f44928924ee460747c5b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:34:55.130334 7836 raft_consensus.cc:3037] T e2a4ba5587bd418899e2cf66f8cf6b2a P ced49025d91f44928924ee460747c5b4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:55.130964 7836 raft_consensus.cc:504] T e2a4ba5587bd418899e2cf66f8cf6b2a P ced49025d91f44928924ee460747c5b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f784af0f7a0463583c819774b174e5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39961 } } peers { permanent_uuid: "ced49025d91f44928924ee460747c5b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33037 } } peers { permanent_uuid: "f7e8161e260e47cbad68b8e313fe23fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46089 } }
I20260928 02:34:55.131075 7836 leader_election.cc:288] T e2a4ba5587bd418899e2cf66f8cf6b2a P ced49025d91f44928924ee460747c5b4 [CANDIDATE]: Term 1 election: Requested vote from peers 2f784af0f7a0463583c819774b174e5c (127.0.98.1:39961), f7e8161e260e47cbad68b8e313fe23fd (127.0.98.2:46089)
I20260928 02:34:55.131206 7423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2a4ba5587bd418899e2cf66f8cf6b2a" candidate_uuid: "ced49025d91f44928924ee460747c5b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f784af0f7a0463583c819774b174e5c"
I20260928 02:34:55.131235 7560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2a4ba5587bd418899e2cf66f8cf6b2a" candidate_uuid: "ced49025d91f44928924ee460747c5b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7e8161e260e47cbad68b8e313fe23fd"
I20260928 02:34:55.131273 7423 raft_consensus.cc:3037] T e2a4ba5587bd418899e2cf66f8cf6b2a P 2f784af0f7a0463583c819774b174e5c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:55.131292 7560 raft_consensus.cc:3037] T e2a4ba5587bd418899e2cf66f8cf6b2a P f7e8161e260e47cbad68b8e313fe23fd [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:55.131922 7560 raft_consensus.cc:2454] T e2a4ba5587bd418899e2cf66f8cf6b2a P f7e8161e260e47cbad68b8e313fe23fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced49025d91f44928924ee460747c5b4 in term 1.
I20260928 02:34:55.131922 7423 raft_consensus.cc:2454] T e2a4ba5587bd418899e2cf66f8cf6b2a P 2f784af0f7a0463583c819774b174e5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ced49025d91f44928924ee460747c5b4 in term 1.
I20260928 02:34:55.132061 7629 leader_election.cc:302] T e2a4ba5587bd418899e2cf66f8cf6b2a P ced49025d91f44928924ee460747c5b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f784af0f7a0463583c819774b174e5c, ced49025d91f44928924ee460747c5b4; no voters:
I20260928 02:34:55.132160 7836 raft_consensus.cc:2781] T e2a4ba5587bd418899e2cf66f8cf6b2a P ced49025d91f44928924ee460747c5b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:55.132283 7836 raft_consensus.cc:686] T e2a4ba5587bd418899e2cf66f8cf6b2a P ced49025d91f44928924ee460747c5b4 [term 1 LEADER]: Becoming Leader. State: Replica: ced49025d91f44928924ee460747c5b4, State: Running, Role: LEADER
I20260928 02:34:55.132325 7836 consensus_queue.cc:237] T e2a4ba5587bd418899e2cf66f8cf6b2a P ced49025d91f44928924ee460747c5b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f784af0f7a0463583c819774b174e5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39961 } } peers { permanent_uuid: "ced49025d91f44928924ee460747c5b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33037 } } peers { permanent_uuid: "f7e8161e260e47cbad68b8e313fe23fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46089 } }
I20260928 02:34:55.132539 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.132453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.98.3:51223)
0928 02:34:55.132483 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:55.132509 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260928 02:34:55.132822 7097 catalog_manager.cc:5495] T e2a4ba5587bd418899e2cf66f8cf6b2a P ced49025d91f44928924ee460747c5b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ced49025d91f44928924ee460747c5b4 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ced49025d91f44928924ee460747c5b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f784af0f7a0463583c819774b174e5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ced49025d91f44928924ee460747c5b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7e8161e260e47cbad68b8e313fe23fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46089 } health_report { overall_health: UNKNOWN } } }
I20260928 02:34:55.143059 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.141159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39932 (local address 127.0.98.3:33037)
0928 02:34:55.141193 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:55.141195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:55.141216 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:34:55.141298 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:55.141299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:55.141310 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:55.141354 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:34:55.141443 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:55.141926 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:55.142598 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:55.142769 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:55.142810 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:55.142903 (+ 93us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:34:55.142926 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:34:55.142955 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:55.142998 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 02:34:55.143011 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 02:34:55.143762 7560 raft_consensus.cc:1261] T e2a4ba5587bd418899e2cf66f8cf6b2a P f7e8161e260e47cbad68b8e313fe23fd [term 1 FOLLOWER]: Refusing update from remote peer ced49025d91f44928924ee460747c5b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:55.143797 7423 raft_consensus.cc:1261] T e2a4ba5587bd418899e2cf66f8cf6b2a P 2f784af0f7a0463583c819774b174e5c [term 1 FOLLOWER]: Refusing update from remote peer ced49025d91f44928924ee460747c5b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:55.143951 7843 consensus_queue.cc:1035] T e2a4ba5587bd418899e2cf66f8cf6b2a P ced49025d91f44928924ee460747c5b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7e8161e260e47cbad68b8e313fe23fd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:55.144042 7836 consensus_queue.cc:1035] T e2a4ba5587bd418899e2cf66f8cf6b2a P ced49025d91f44928924ee460747c5b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f784af0f7a0463583c819774b174e5c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:55.144613 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.144511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.98.2:58025)
0928 02:34:55.144559 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:55.144586 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 02:34:55.145038 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.144822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.98.1:51299)
0928 02:34:55.144982 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:55.145011 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260928 02:34:55.145550 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.145465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.98.3:36243)
0928 02:34:55.145500 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:55.145523 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 02:34:55.146054 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.145854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.98.1:60611)
0928 02:34:55.145999 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:55.146029 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260928 02:34:55.146063 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.145931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.98.3:60709)
0928 02:34:55.146024 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:55.146045 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 02:34:55.146231 7848 mvcc.cc:205] Tried to move back new op lower bound from 7334145618507464704 to 7334145618462523392. Current Snapshot: MvccSnapshot[applied={T|T < 7334145618507464704}]
I20260928 02:34:55.146382 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.146277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.98.2:36879)
0928 02:34:55.146337 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:55.146359 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 02:34:55.147730 7097 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:34:55.147801 7097 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:34:55.148824 7097 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37118a91babb433594ad029895cee4a7: Sending DeleteTablet for 3 replicas of tablet e2a4ba5587bd418899e2cf66f8cf6b2a
I20260928 02:34:55.149086 7403 tablet_service.cc:1514] Processing DeleteTablet for tablet e2a4ba5587bd418899e2cf66f8cf6b2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:55 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:41858
I20260928 02:34:55.149109 7676 tablet_service.cc:1514] Processing DeleteTablet for tablet e2a4ba5587bd418899e2cf66f8cf6b2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:55 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:39928
I20260928 02:34:55.149329 7540 tablet_service.cc:1514] Processing DeleteTablet for tablet e2a4ba5587bd418899e2cf66f8cf6b2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:55 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:55070
I20260928 02:34:55.149355 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 7252
I20260928 02:34:55.150106 7857 tablet_replica.cc:330] T e2a4ba5587bd418899e2cf66f8cf6b2a P f7e8161e260e47cbad68b8e313fe23fd: stopping tablet replica
I20260928 02:34:55.150159 7857 raft_consensus.cc:2229] T e2a4ba5587bd418899e2cf66f8cf6b2a P f7e8161e260e47cbad68b8e313fe23fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:34:55.150202 7857 raft_consensus.cc:2258] T e2a4ba5587bd418899e2cf66f8cf6b2a P f7e8161e260e47cbad68b8e313fe23fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:55.150460 7857 ts_tablet_manager.cc:1905] T e2a4ba5587bd418899e2cf66f8cf6b2a P f7e8161e260e47cbad68b8e313fe23fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:55.151695 7857 ts_tablet_manager.cc:1918] T e2a4ba5587bd418899e2cf66f8cf6b2a P f7e8161e260e47cbad68b8e313fe23fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:34:55.151742 7857 log.cc:1196] T e2a4ba5587bd418899e2cf66f8cf6b2a P f7e8161e260e47cbad68b8e313fe23fd: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-1/wal/wals/e2a4ba5587bd418899e2cf66f8cf6b2a
I20260928 02:34:55.151961 7857 ts_tablet_manager.cc:1939] T e2a4ba5587bd418899e2cf66f8cf6b2a P f7e8161e260e47cbad68b8e313fe23fd: Deleting consensus metadata
I20260928 02:34:55.152209 7084 catalog_manager.cc:4841] TS f7e8161e260e47cbad68b8e313fe23fd (127.0.98.2:46089): tablet e2a4ba5587bd418899e2cf66f8cf6b2a (table test-table [id=307c41b28a8f4d8f9814ec3fdbfd9edc]) successfully deleted
I20260928 02:34:55.153339 7856 tablet_replica.cc:330] T e2a4ba5587bd418899e2cf66f8cf6b2a P ced49025d91f44928924ee460747c5b4: stopping tablet replica
I20260928 02:34:55.153391 7856 raft_consensus.cc:2229] T e2a4ba5587bd418899e2cf66f8cf6b2a P ced49025d91f44928924ee460747c5b4 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:34:55.153465 7856 raft_consensus.cc:2258] T e2a4ba5587bd418899e2cf66f8cf6b2a P ced49025d91f44928924ee460747c5b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:55.153678 7856 ts_tablet_manager.cc:1905] T e2a4ba5587bd418899e2cf66f8cf6b2a P ced49025d91f44928924ee460747c5b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:55.154381 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:55.154278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39961 (local address 127.0.0.1:41864)
0928 02:34:55.154328 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:55.154353 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 02:34:55.154393 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 7477
W20260928 02:34:55.154512 7082 catalog_manager.cc:4584] TS 2f784af0f7a0463583c819774b174e5c (127.0.98.1:39961): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e2a4ba5587bd418899e2cf66f8cf6b2a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39961: connect: Connection refused (error 111)
I20260928 02:34:55.156206 7856 ts_tablet_manager.cc:1918] T e2a4ba5587bd418899e2cf66f8cf6b2a P ced49025d91f44928924ee460747c5b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:34:55.156298 7856 log.cc:1196] T e2a4ba5587bd418899e2cf66f8cf6b2a P ced49025d91f44928924ee460747c5b4: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790562862804661-392-0/minicluster-data/ts-2/wal/wals/e2a4ba5587bd418899e2cf66f8cf6b2a
I20260928 02:34:55.156490 7856 ts_tablet_manager.cc:1939] T e2a4ba5587bd418899e2cf66f8cf6b2a P ced49025d91f44928924ee460747c5b4: Deleting consensus metadata
I20260928 02:34:55.156808 7082 catalog_manager.cc:4841] TS ced49025d91f44928924ee460747c5b4 (127.0.98.3:33037): tablet e2a4ba5587bd418899e2cf66f8cf6b2a (table test-table [id=307c41b28a8f4d8f9814ec3fdbfd9edc]) successfully deleted
I20260928 02:34:55.159124 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 7613
I20260928 02:34:55.164181 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 6964
I20260928 02:34:55.167784 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 7152
2026-09-28T02:34:55Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4344 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:34:55 dist-test-slave-fz47 krb5kdc[7860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:34:55 dist-test-slave-fz47 krb5kdc[7860](info): set up 2 sockets
Sep 28 02:34:55 dist-test-slave-fz47 krb5kdc[7860](info): commencing operation
krb5kdc: starting...
W20260928 02:34:56.109587 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:34:56 dist-test-slave-fz47 krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:34:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:34:56Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260928 02:34:56.219360 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35019
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:56.296494 7873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:56.296680 7873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:56.296707 7873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:56.297890 7873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:34:56.297932 7873 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:34:56.297948 7873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:56.297966 7873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:56.297981 7873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:34:56.297996 7873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:34:56.299393 7873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35019
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:56.299613 7873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:56.299813 7873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:56.304186 7878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:56.304332 7873 server_base.cc:1034] running on GCE node
W20260928 02:34:56.304318 7879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:56.304243 7881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:56.304623 7873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:56.304900 7873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:56.306046 7873 hybrid_clock.cc:648] HybridClock initialized: now 1790562896306030 us; error 26 us; skew 500 ppm
Sep 28 02:34:56 dist-test-slave-fz47 krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562896, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:56.308449 7873 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260928 02:34:56.308868 7873 webserver.cc:460] Webserver started at http://127.0.98.62:38027/ using document root <none> and password file <none>
I20260928 02:34:56.309062 7873 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:56.309106 7873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:56.309218 7873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:56.310170 7873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "ca69e07373f44b6096c5c7cd6b96bdd5"
format_stamp: "Formatted at 2026-09-28 02:34:56 on dist-test-slave-fz47"
server_key: "c623e5c122d9265a9cc7df3c0456cf93"
server_key_iv: "eb2a2f0dfbdd010b65fcb36151f40418"
server_key_version: "encryptionkey@0"
I20260928 02:34:56.310475 7873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "ca69e07373f44b6096c5c7cd6b96bdd5"
format_stamp: "Formatted at 2026-09-28 02:34:56 on dist-test-slave-fz47"
server_key: "c623e5c122d9265a9cc7df3c0456cf93"
server_key_iv: "eb2a2f0dfbdd010b65fcb36151f40418"
server_key_version: "encryptionkey@0"
I20260928 02:34:56.311628 7873 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:34:56.312282 7888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:56.312463 7873 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:56.312559 7873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "ca69e07373f44b6096c5c7cd6b96bdd5"
format_stamp: "Formatted at 2026-09-28 02:34:56 on dist-test-slave-fz47"
server_key: "c623e5c122d9265a9cc7df3c0456cf93"
server_key_iv: "eb2a2f0dfbdd010b65fcb36151f40418"
server_key_version: "encryptionkey@0"
I20260928 02:34:56.312618 7873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:56.335855 7873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:56.337648 7873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:56.337817 7873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:56.342250 7873 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35019
I20260928 02:34:56.342295 7950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35019 every 8 connection(s)
I20260928 02:34:56.342590 7873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:34:56.343171 7951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:56.343487 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 7873
I20260928 02:34:56.343580 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:34:56.343746 392 external_mini_cluster.cc:1428] Setting key ec09cfeb08f30c70b6edf5162e7ce5b9
I20260928 02:34:56.345985 7951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca69e07373f44b6096c5c7cd6b96bdd5: Bootstrap starting.
I20260928 02:34:56.346642 7951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca69e07373f44b6096c5c7cd6b96bdd5: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:56.346933 7951 log.cc:824] T 00000000000000000000000000000000 P ca69e07373f44b6096c5c7cd6b96bdd5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:56.347615 7951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca69e07373f44b6096c5c7cd6b96bdd5: No bootstrap required, opened a new log
I20260928 02:34:56.349786 7951 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca69e07373f44b6096c5c7cd6b96bdd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca69e07373f44b6096c5c7cd6b96bdd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35019 } }
I20260928 02:34:56.349922 7951 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca69e07373f44b6096c5c7cd6b96bdd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:56.349948 7951 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca69e07373f44b6096c5c7cd6b96bdd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca69e07373f44b6096c5c7cd6b96bdd5, State: Initialized, Role: FOLLOWER
I20260928 02:34:56.350082 7951 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca69e07373f44b6096c5c7cd6b96bdd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca69e07373f44b6096c5c7cd6b96bdd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35019 } }
I20260928 02:34:56.350160 7951 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca69e07373f44b6096c5c7cd6b96bdd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:34:56.350196 7951 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca69e07373f44b6096c5c7cd6b96bdd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:34:56.350241 7951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca69e07373f44b6096c5c7cd6b96bdd5 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 02:34:56 dist-test-slave-fz47 krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 02:34:56.350927 7951 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca69e07373f44b6096c5c7cd6b96bdd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca69e07373f44b6096c5c7cd6b96bdd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35019 } }
I20260928 02:34:56.351034 7951 leader_election.cc:302] T 00000000000000000000000000000000 P ca69e07373f44b6096c5c7cd6b96bdd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca69e07373f44b6096c5c7cd6b96bdd5; no voters:
I20260928 02:34:56.351186 7951 leader_election.cc:288] T 00000000000000000000000000000000 P ca69e07373f44b6096c5c7cd6b96bdd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:34:56.351239 7956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca69e07373f44b6096c5c7cd6b96bdd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:56.351434 7951 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca69e07373f44b6096c5c7cd6b96bdd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:34:56.351734 7956 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca69e07373f44b6096c5c7cd6b96bdd5 [term 1 LEADER]: Becoming Leader. State: Replica: ca69e07373f44b6096c5c7cd6b96bdd5, State: Running, Role: LEADER
I20260928 02:34:56.351897 7956 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca69e07373f44b6096c5c7cd6b96bdd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca69e07373f44b6096c5c7cd6b96bdd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35019 } }
I20260928 02:34:56.352516 7958 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca69e07373f44b6096c5c7cd6b96bdd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca69e07373f44b6096c5c7cd6b96bdd5. Latest consensus state: current_term: 1 leader_uuid: "ca69e07373f44b6096c5c7cd6b96bdd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca69e07373f44b6096c5c7cd6b96bdd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35019 } } }
I20260928 02:34:56.352591 7958 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca69e07373f44b6096c5c7cd6b96bdd5 [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:56.352846 7971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:34:56.352995 7957 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca69e07373f44b6096c5c7cd6b96bdd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca69e07373f44b6096c5c7cd6b96bdd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca69e07373f44b6096c5c7cd6b96bdd5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35019 } } }
I20260928 02:34:56.353384 7957 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca69e07373f44b6096c5c7cd6b96bdd5 [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:56.353622 7971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:34:56.354658 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:56.344220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38262 (local address 127.0.98.62:35019)
0928 02:34:56.344412 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:56.344422 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:56.344455 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:34:56.344930 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:56.344939 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:56.345000 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:56.345143 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:56.345735 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.347021 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.347824 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.348440 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.348559 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:56.351095 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:56.351100 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:56.351100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:56.351113 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:56.353178 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:56.353567 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:56.353570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:56.353571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:56.353674 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:56.353874 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:56.353875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:56.353875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:56.354094 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:56.354222 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:56.354338 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 02:34:56.354463 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":72,"threads_started":1}
I20260928 02:34:56.355069 7971 catalog_manager.cc:1347] Generated new cluster ID: 47e604b21dc74279885c64a2f56139fa
I20260928 02:34:56.355113 7971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:34:56.366346 7971 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:34:56.366828 7971 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:34:56.377146 7971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca69e07373f44b6096c5c7cd6b96bdd5: Generated new TSK 0
I20260928 02:34:56.377334 7971 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260928 02:34:56.403693 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35019
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:56.480983 7977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:56.481168 7977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:56.481190 7977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:56.482301 7977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:56.482342 7977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:56.482360 7977 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:34:56.482376 7977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:34:56.483749 7977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:56.483930 7977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:56.484092 7977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:56.488482 7985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:56.488489 7983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:56.488627 7977 server_base.cc:1034] running on GCE node
W20260928 02:34:56.488538 7982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:56.488934 7977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:56.489116 7977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:56.490242 7977 hybrid_clock.cc:648] HybridClock initialized: now 1790562896490228 us; error 29 us; skew 500 ppm
Sep 28 02:34:56 dist-test-slave-fz47 krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562896, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:56.492455 7977 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 02:34:56.492826 7977 webserver.cc:460] Webserver started at http://127.0.98.1:41603/ using document root <none> and password file <none>
I20260928 02:34:56.493010 7977 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:56.493052 7977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:56.493160 7977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:56.494151 7977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-0/data/instance:
uuid: "785d3ca35dcd4aea838abdfdf3584460"
format_stamp: "Formatted at 2026-09-28 02:34:56 on dist-test-slave-fz47"
server_key: "4737be86fe8c61ac8ddd8117f2736ebf"
server_key_iv: "61f8b33ab340618e3127374364bd6876"
server_key_version: "encryptionkey@0"
I20260928 02:34:56.494464 7977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-0/wal/instance:
uuid: "785d3ca35dcd4aea838abdfdf3584460"
format_stamp: "Formatted at 2026-09-28 02:34:56 on dist-test-slave-fz47"
server_key: "4737be86fe8c61ac8ddd8117f2736ebf"
server_key_iv: "61f8b33ab340618e3127374364bd6876"
server_key_version: "encryptionkey@0"
I20260928 02:34:56.495668 7977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:34:56.496282 7992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:56.496425 7977 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:56.496495 7977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "785d3ca35dcd4aea838abdfdf3584460"
format_stamp: "Formatted at 2026-09-28 02:34:56 on dist-test-slave-fz47"
server_key: "4737be86fe8c61ac8ddd8117f2736ebf"
server_key_iv: "61f8b33ab340618e3127374364bd6876"
server_key_version: "encryptionkey@0"
I20260928 02:34:56.496556 7977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:56.504828 7977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:56.506433 7977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:56.506542 7977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:56.506951 7977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:56.506989 7977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:56.507007 7977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:56.507020 7977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:34:56 dist-test-slave-fz47 krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562896, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 02:34:56.513435 7977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43567
I20260928 02:34:56.513484 8107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43567 every 8 connection(s)
I20260928 02:34:56.513795 7977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
I20260928 02:34:56.515596 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:56.507339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48387 (local address 127.0.98.62:35019)
0928 02:34:56.507398 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:56.507400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:56.507542 (+ 142us) server_negotiation.cc:407] Connection header received
0928 02:34:56.508016 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:56.508017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:56.508029 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:56.508100 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:56.508813 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.509587 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.510583 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.510910 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.510980 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:56.512962 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:56.512963 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:56.512963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:56.512970 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:56.514392 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:56.514757 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:56.514758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:56.514759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:56.514812 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:56.515011 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:56.515011 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:56.515012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:56.515134 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:56.515188 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:56.515430 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 02:34:56.515509 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 02:34:56.515738 8001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:56.507294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35019 (local address 127.0.98.1:48387)
0928 02:34:56.507436 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:56.507474 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:56.507930 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:56.508164 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:56.508168 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:56.508451 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:56.508706 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:56.508711 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.509678 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.509679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:56.510503 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:56.510506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.511000 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.511001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:56.511124 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:56.511556 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:56.511564 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:56.512880 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:56.514474 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:56.514480 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:56.514482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:56.514695 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:56.514867 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:56.514868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:56.514868 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:56.514958 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:56.515186 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:56.515193 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:56.515313 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 02:34:56.515404 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 02:34:56.515546 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260928 02:34:56.516235 8108 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35019
I20260928 02:34:56.516314 8108 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:56.516469 8108 heartbeater.cc:511] Master 127.0.98.62:35019 requested a full tablet report, sending...
I20260928 02:34:56.516937 7905 ts_manager.cc:194] Registered new tserver with Master: 785d3ca35dcd4aea838abdfdf3584460 (127.0.98.1:43567)
I20260928 02:34:56.517675 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 7977
I20260928 02:34:56.517763 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-0/wal/instance
I20260928 02:34:56.517853 7905 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48387
I20260928 02:34:56.517951 392 external_mini_cluster.cc:1428] Setting key 6d1d94acd4a64b86a7f7ab3dd8594495
I20260928 02:34:56.522733 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:56.517139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49833 (local address 127.0.98.62:35019)
0928 02:34:56.517180 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:56.517181 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:56.517288 (+ 107us) server_negotiation.cc:407] Connection header received
0928 02:34:56.517352 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:56.517352 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:56.517361 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:56.517404 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:56.517824 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.518537 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.519343 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.519595 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.519668 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:56.520496 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:56.520497 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:56.520498 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:56.520503 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:56.521803 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:56.522136 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:56.522136 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:56.522136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:56.522177 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:56.522362 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:56.522363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:56.522363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:56.522446 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:56.522486 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:56.522627 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 02:34:56.522676 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":106,"server-negotiator.queue_time_us":20}
I20260928 02:34:56.522799 8114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:56.517093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35019 (local address 127.0.98.1:49833)
0928 02:34:56.517218 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:56.517229 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:56.517306 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:56.517409 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:56.517410 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:56.517612 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:56.517668 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:56.517670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.518614 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.518615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:56.519263 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:56.519264 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.519675 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.519675 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:56.519740 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:56.520073 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:56.520077 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:56.520410 (+ 333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:56.521878 (+ 1468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:56.521879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:56.521879 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:56.522072 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:56.522243 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:56.522243 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:56.522244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:56.522305 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:56.522493 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:56.522493 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:56.522535 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 02:34:56.522609 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 02:34:56.522698 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":24,"thread_start_us":66,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260928 02:34:56.539415 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35019
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:56.616230 8117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:56.616447 8117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:56.616473 8117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:56.617628 8117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:56.617671 8117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:56.617690 8117 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:34:56.617707 8117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:34:56.619122 8117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:56.619323 8117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:56.619513 8117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:56.623992 8122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:56.623973 8125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:56.623989 8123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:56.624219 8117 server_base.cc:1034] running on GCE node
I20260928 02:34:56.624406 8117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:56.624580 8117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:56.625702 8117 hybrid_clock.cc:648] HybridClock initialized: now 1790562896625688 us; error 28 us; skew 500 ppm
Sep 28 02:34:56 dist-test-slave-fz47 krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562896, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:56.627844 8117 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 02:34:56.628224 8117 webserver.cc:460] Webserver started at http://127.0.98.2:46853/ using document root <none> and password file <none>
I20260928 02:34:56.628424 8117 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:56.628468 8117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:56.628561 8117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:56.629570 8117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-1/data/instance:
uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5"
format_stamp: "Formatted at 2026-09-28 02:34:56 on dist-test-slave-fz47"
server_key: "dd92836cfa21062c67584d79d1b0d452"
server_key_iv: "698ee99a86a2d9f9b30b6e8aea760857"
server_key_version: "encryptionkey@0"
I20260928 02:34:56.629878 8117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5"
format_stamp: "Formatted at 2026-09-28 02:34:56 on dist-test-slave-fz47"
server_key: "dd92836cfa21062c67584d79d1b0d452"
server_key_iv: "698ee99a86a2d9f9b30b6e8aea760857"
server_key_version: "encryptionkey@0"
I20260928 02:34:56.631048 8117 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:34:56.631742 8132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:56.631901 8117 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:56.631973 8117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5"
format_stamp: "Formatted at 2026-09-28 02:34:56 on dist-test-slave-fz47"
server_key: "dd92836cfa21062c67584d79d1b0d452"
server_key_iv: "698ee99a86a2d9f9b30b6e8aea760857"
server_key_version: "encryptionkey@0"
I20260928 02:34:56.632032 8117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:56.658267 8117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:56.660048 8117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:56.660162 8117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:56.660669 8117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:56.660709 8117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:56.660737 8117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:56.660755 8117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:34:56 dist-test-slave-fz47 krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562896, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 02:34:56.667140 8117 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42811
I20260928 02:34:56.667466 8117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
I20260928 02:34:56.667430 8247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42811 every 8 connection(s)
I20260928 02:34:56.669373 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:56.661035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39421 (local address 127.0.98.62:35019)
0928 02:34:56.661092 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:56.661094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:56.661281 (+ 187us) server_negotiation.cc:407] Connection header received
0928 02:34:56.661775 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:56.661776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:56.661788 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:56.661867 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:56.662449 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.663209 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.664183 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.664474 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.664549 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:56.667004 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:56.667006 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:56.667006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:56.667013 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:56.668331 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:56.668649 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:56.668650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:56.668650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:56.668692 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:56.668880 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:56.668880 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:56.668881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:56.668968 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:56.669010 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:56.669220 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 02:34:56.669300 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 02:34:56.669528 8141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:56.661014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35019 (local address 127.0.98.2:39421)
0928 02:34:56.661157 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:56.661195 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:56.661692 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:56.661938 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:56.661942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:56.662182 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:56.662382 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:56.662386 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.663289 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.663290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:56.664121 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:56.664123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.664569 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.664570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:56.664695 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:56.665104 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:56.665112 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:56.666915 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:56.668403 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:56.668407 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:56.668408 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:56.668598 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:56.668734 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:56.668734 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:56.668735 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:56.668815 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:56.669006 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:56.669009 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:56.669111 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 02:34:56.669198 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 02:34:56.669348 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
I20260928 02:34:56.669934 8248 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35019
I20260928 02:34:56.670007 8248 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:56.670161 8248 heartbeater.cc:511] Master 127.0.98.62:35019 requested a full tablet report, sending...
I20260928 02:34:56.670517 7905 ts_manager.cc:194] Registered new tserver with Master: ee5ade571f0349a1b4e3fa4edbeae2e5 (127.0.98.2:42811)
I20260928 02:34:56.670961 7905 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39421
I20260928 02:34:56.673858 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 8117
I20260928 02:34:56.673974 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-1/wal/instance
I20260928 02:34:56.674165 392 external_mini_cluster.cc:1428] Setting key f7b8a946d00b2c064d726753fb9afe78
I20260928 02:34:56.675539 8254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:56.670797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35019 (local address 127.0.98.2:49991)
0928 02:34:56.670906 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:56.670915 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:56.670969 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:56.671115 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:56.671116 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:56.671303 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:56.671355 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:56.671357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.671935 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.671936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:56.672426 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:56.672427 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.672686 (+ 259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.672687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:56.672725 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:56.673083 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:56.673087 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:56.673376 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:56.674694 (+ 1318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:56.674694 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:56.674695 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:56.674872 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:56.675040 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:56.675040 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:56.675041 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:56.675084 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:56.675335 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:56.675336 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:56.675364 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 02:34:56.675408 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 02:34:56.675473 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":61,"threads_started":1}
I20260928 02:34:56.675567 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:56.670852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49991 (local address 127.0.98.62:35019)
0928 02:34:56.670889 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:56.670890 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:56.670955 (+ 65us) server_negotiation.cc:407] Connection header received
0928 02:34:56.671044 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:56.671045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:56.671055 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:56.671109 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:56.671424 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.671863 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.672481 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.672643 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.672683 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:56.673441 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:56.673441 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:56.673441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:56.673446 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:56.674628 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:56.674935 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:56.674936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:56.674936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:56.674987 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:56.675128 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:56.675128 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:56.675129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:56.675286 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:56.675346 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:56.675430 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 02:34:56.675493 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260928 02:34:56.696450 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35019
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:56.773586 8257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:56.773778 8257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:56.773800 8257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:56.774860 8257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:56.774891 8257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:56.774909 8257 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:34:56.774924 8257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:34:56.776348 8257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:56.776526 8257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:56.776683 8257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:56.781407 8263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:56.781454 8257 server_base.cc:1034] running on GCE node
W20260928 02:34:56.781431 8265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:56.781421 8262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:56.781767 8257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:56.781975 8257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:56.783110 8257 hybrid_clock.cc:648] HybridClock initialized: now 1790562896783098 us; error 28 us; skew 500 ppm
Sep 28 02:34:56 dist-test-slave-fz47 krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562896, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:56.785326 8257 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 02:34:56.785734 8257 webserver.cc:460] Webserver started at http://127.0.98.3:35059/ using document root <none> and password file <none>
I20260928 02:34:56.785938 8257 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:56.785982 8257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:56.786100 8257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:56.787088 8257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-2/data/instance:
uuid: "a1187d9ed1b24dac9b4887d60838afb9"
format_stamp: "Formatted at 2026-09-28 02:34:56 on dist-test-slave-fz47"
server_key: "5c4e89dfbbbaff6ca72c9680361b9efd"
server_key_iv: "ae0824422f08d21f31497ddee0e5e218"
server_key_version: "encryptionkey@0"
I20260928 02:34:56.787397 8257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a1187d9ed1b24dac9b4887d60838afb9"
format_stamp: "Formatted at 2026-09-28 02:34:56 on dist-test-slave-fz47"
server_key: "5c4e89dfbbbaff6ca72c9680361b9efd"
server_key_iv: "ae0824422f08d21f31497ddee0e5e218"
server_key_version: "encryptionkey@0"
I20260928 02:34:56.788539 8257 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:34:56.789234 8272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:56.789397 8257 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:56.789472 8257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "a1187d9ed1b24dac9b4887d60838afb9"
format_stamp: "Formatted at 2026-09-28 02:34:56 on dist-test-slave-fz47"
server_key: "5c4e89dfbbbaff6ca72c9680361b9efd"
server_key_iv: "ae0824422f08d21f31497ddee0e5e218"
server_key_version: "encryptionkey@0"
I20260928 02:34:56.789533 8257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:56.809834 8257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:56.811787 8257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:56.811915 8257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:56.812525 8257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:34:56.812568 8257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:56.812603 8257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:34:56.812618 8257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:56.819275 8257 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35633
I20260928 02:34:56.819350 8387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35633 every 8 connection(s)
Sep 28 02:34:56 dist-test-slave-fz47 krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562896, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 02:34:56.819610 8257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
I20260928 02:34:56.820638 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 8257
I20260928 02:34:56.820792 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-2/wal/instance
I20260928 02:34:56.821002 392 external_mini_cluster.cc:1428] Setting key 7664a3f59190d5468d06bcaa1c31b4d7
I20260928 02:34:56.822412 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:56.813971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48503 (local address 127.0.98.62:35019)
0928 02:34:56.814027 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:56.814030 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:56.814261 (+ 231us) server_negotiation.cc:407] Connection header received
0928 02:34:56.814752 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:56.814753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:56.814765 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:56.814827 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:56.815561 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.816198 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.817360 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.817583 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.817640 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:56.819881 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:56.819882 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:56.819883 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:56.819889 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:56.821286 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:56.821703 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:56.821704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:56.821705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:56.821763 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:56.821978 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:56.821978 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:56.821978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:56.822072 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:56.822118 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:56.822300 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 02:34:56.822353 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 02:34:56.822570 8283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:56.813951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35019 (local address 127.0.98.3:48503)
0928 02:34:56.814115 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:56.814153 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:56.814652 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:56.814932 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:56.814935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:56.815235 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:56.815478 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:56.815483 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.816279 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.816280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:56.817292 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:56.817295 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.817651 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.817652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:56.817777 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:56.818248 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:56.818258 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:56.819781 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:56.821361 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:56.821366 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:56.821368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:56.821640 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:56.821822 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:56.821823 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:56.821823 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:56.821931 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:56.822109 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:56.822111 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:56.822211 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 02:34:56.822288 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 02:34:56.822394 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
I20260928 02:34:56.822979 8388 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35019
I20260928 02:34:56.823058 8388 heartbeater.cc:464] Registering TS with master...
I20260928 02:34:56.823232 8388 heartbeater.cc:511] Master 127.0.98.62:35019 requested a full tablet report, sending...
I20260928 02:34:56.823555 7902 ts_manager.cc:194] Registered new tserver with Master: a1187d9ed1b24dac9b4887d60838afb9 (127.0.98.3:35633)
I20260928 02:34:56.823921 7902 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48503
I20260928 02:34:56.828315 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:56.823955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33615 (local address 127.0.98.62:35019)
0928 02:34:56.823988 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:56.823990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:56.824136 (+ 146us) server_negotiation.cc:407] Connection header received
0928 02:34:56.824200 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:56.824201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:56.824211 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:56.824263 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:56.824500 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.824989 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.825681 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.825863 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.825913 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:56.826606 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:56.826607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:56.826607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:56.826611 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:56.827548 (+ 937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:56.827850 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:56.827850 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:56.827850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:56.827890 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:56.828016 (+ 126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:56.828017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:56.828017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:56.828094 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:56.828133 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:56.828216 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 02:34:56.828264 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 02:34:56.828327 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:56.823884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35019 (local address 127.0.98.3:33615)
0928 02:34:56.824086 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:56.824097 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:56.824147 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:56.824263 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:56.824264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:56.824396 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:56.824441 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:56.824442 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.825059 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.825060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:56.825600 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:56.825602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.825916 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.825917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:56.825959 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:56.826315 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:56.826319 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:56.826549 (+ 230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:56.827629 (+ 1080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:56.827630 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:56.827630 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:56.827783 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:56.827935 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:56.827935 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:56.827936 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:56.827979 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:56.828127 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:56.828128 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:56.828156 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 02:34:56.828198 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 02:34:56.828257 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":55,"threads_started":1}
I20260928 02:34:56.832453 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:34:56 dist-test-slave-fz47 krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562896, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:34:56 dist-test-slave-fz47 krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:34:56 dist-test-slave-fz47 krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:34:56.851497 7954 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:34:56.846583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38272 (local address 127.0.98.62:35019)
0928 02:34:56.846629 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:56.846631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:56.846640 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:34:56.846757 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:56.846758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:56.846767 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:56.846810 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:56.847312 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.847794 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.848485 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.848660 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.848712 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:56.851396 (+ 2684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38272: BlockingRecv error: recv got EOF from 127.0.0.1:38272 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 28 02:34:56 dist-test-slave-fz47 krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562896, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 02:34:56.858065 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:56.852226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38288 (local address 127.0.98.62:35019)
0928 02:34:56.852275 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:56.852277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:56.852317 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:34:56.852427 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:56.852428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:56.852436 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:56.852485 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:56.852916 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.853437 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.854078 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.854270 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.854321 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:56.856135 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:56.856136 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:56.856137 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:56.856143 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:56.857115 (+ 972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:56.857436 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:56.857437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:56.857438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:56.857487 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:56.857639 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:56.857640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:56.857640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:56.857744 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:56.857824 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:56.857917 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 02:34:56.857983 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 02:34:56.859350 7902 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:34:56.860837 7902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:34:56.866571 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:56.862686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35633 (local address 127.0.0.1:32856)
0928 02:34:56.862823 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:56.862859 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:56.862915 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:56.863361 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:56.863363 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:56.863366 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:56.863427 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:56.863428 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.864374 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.864375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:56.865526 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:56.865529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.866301 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.866302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:56.866374 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:56.866387 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 02:34:56.866454 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 02:34:56.866501 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":95,"threads_started":1}
I20260928 02:34:56.866586 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:56.863029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32856 (local address 127.0.98.3:35633)
0928 02:34:56.863172 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:56.863176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:56.863187 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:34:56.863247 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:56.863249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:56.863265 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:56.863346 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:56.863511 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.864300 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.865614 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.866232 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.866291 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:56.866371 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:56.866469 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 02:34:56.866501 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260928 02:34:56.866863 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:56.862708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42811 (local address 127.0.0.1:40210)
0928 02:34:56.863153 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:56.863162 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:56.863208 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:56.863463 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:56.863464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:56.863466 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:56.863518 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:56.863519 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.864768 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.864769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:56.865842 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:56.865844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.866653 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.866654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:56.866722 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:56.866726 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:34:56.866755 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 02:34:56.866792 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":53,"threads_started":1}
I20260928 02:34:56.866935 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:56.862913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43567 (local address 127.0.0.1:47682)
0928 02:34:56.863350 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:56.863359 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:56.863405 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:56.863685 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:56.863686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:56.863687 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:56.863744 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:56.863746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.864911 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.864912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:56.865764 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:56.865765 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.866733 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.866734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:56.866791 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:56.866794 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:34:56.866852 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 02:34:56.866881 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":1,"thread_start_us":54,"threads_started":1}
I20260928 02:34:56.867013 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:56.863007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47682 (local address 127.0.98.1:43567)
0928 02:34:56.863148 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:56.863152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:56.863462 (+ 310us) server_negotiation.cc:407] Connection header received
0928 02:34:56.863505 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:56.863507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:56.863543 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:56.863608 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:56.863817 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.864846 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.865836 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.866558 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.866625 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:56.866693 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:56.866877 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 02:34:56.866919 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
I20260928 02:34:56.867266 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:56.863195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40210 (local address 127.0.98.2:42811)
0928 02:34:56.863329 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:56.863332 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:56.863341 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:34:56.863373 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:56.863375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:56.863389 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:56.863465 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:56.863599 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.864685 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.865907 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:56.866558 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:56.867031 (+ 473us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:56.867106 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:56.867133 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 02:34:56.867170 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
I20260928 02:34:56.867492 8322 tablet_service.cc:1467] Processing CreateTablet for tablet d35fe4ff3d474b66b06a2bef87dfbfdd (DEFAULT_TABLE table=test-table [id=47ed87d2157545f0beb4d4e4f2d86163]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:56.867616 8042 tablet_service.cc:1467] Processing CreateTablet for tablet d35fe4ff3d474b66b06a2bef87dfbfdd (DEFAULT_TABLE table=test-table [id=47ed87d2157545f0beb4d4e4f2d86163]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:56.867849 8322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d35fe4ff3d474b66b06a2bef87dfbfdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:56.867909 8042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d35fe4ff3d474b66b06a2bef87dfbfdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:56.869844 8182 tablet_service.cc:1467] Processing CreateTablet for tablet d35fe4ff3d474b66b06a2bef87dfbfdd (DEFAULT_TABLE table=test-table [id=47ed87d2157545f0beb4d4e4f2d86163]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:56.870153 8182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d35fe4ff3d474b66b06a2bef87dfbfdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:56.870823 8413 tablet_bootstrap.cc:492] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460: Bootstrap starting.
I20260928 02:34:56.870970 8414 tablet_bootstrap.cc:492] T d35fe4ff3d474b66b06a2bef87dfbfdd P a1187d9ed1b24dac9b4887d60838afb9: Bootstrap starting.
I20260928 02:34:56.871392 8413 tablet_bootstrap.cc:654] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:56.871536 8414 tablet_bootstrap.cc:654] T d35fe4ff3d474b66b06a2bef87dfbfdd P a1187d9ed1b24dac9b4887d60838afb9: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:56.871675 8413 log.cc:824] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:56.871855 8414 log.cc:824] T d35fe4ff3d474b66b06a2bef87dfbfdd P a1187d9ed1b24dac9b4887d60838afb9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:56.872291 8413 tablet_bootstrap.cc:492] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460: No bootstrap required, opened a new log
I20260928 02:34:56.872342 8413 ts_tablet_manager.cc:1397] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:34:56.873231 8417 tablet_bootstrap.cc:492] T d35fe4ff3d474b66b06a2bef87dfbfdd P ee5ade571f0349a1b4e3fa4edbeae2e5: Bootstrap starting.
I20260928 02:34:56.873795 8417 tablet_bootstrap.cc:654] T d35fe4ff3d474b66b06a2bef87dfbfdd P ee5ade571f0349a1b4e3fa4edbeae2e5: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:56.873881 8413 raft_consensus.cc:348] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } }
I20260928 02:34:56.874042 8413 raft_consensus.cc:374] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:56.874081 8413 raft_consensus.cc:729] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 785d3ca35dcd4aea838abdfdf3584460, State: Initialized, Role: FOLLOWER
I20260928 02:34:56.874099 8417 log.cc:824] T d35fe4ff3d474b66b06a2bef87dfbfdd P ee5ade571f0349a1b4e3fa4edbeae2e5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:56.874213 8413 consensus_queue.cc:260] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } }
I20260928 02:34:56.874461 8413 ts_tablet_manager.cc:1428] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:34:56.874490 8108 heartbeater.cc:503] Master 127.0.98.62:35019 was elected leader, sending a full tablet report...
I20260928 02:34:56.874747 8417 tablet_bootstrap.cc:492] T d35fe4ff3d474b66b06a2bef87dfbfdd P ee5ade571f0349a1b4e3fa4edbeae2e5: No bootstrap required, opened a new log
I20260928 02:34:56.874812 8417 ts_tablet_manager.cc:1397] T d35fe4ff3d474b66b06a2bef87dfbfdd P ee5ade571f0349a1b4e3fa4edbeae2e5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 02:34:56.875007 8414 tablet_bootstrap.cc:492] T d35fe4ff3d474b66b06a2bef87dfbfdd P a1187d9ed1b24dac9b4887d60838afb9: No bootstrap required, opened a new log
I20260928 02:34:56.875070 8414 ts_tablet_manager.cc:1397] T d35fe4ff3d474b66b06a2bef87dfbfdd P a1187d9ed1b24dac9b4887d60838afb9: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 02:34:56.876359 8417 raft_consensus.cc:348] T d35fe4ff3d474b66b06a2bef87dfbfdd P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } }
I20260928 02:34:56.876473 8417 raft_consensus.cc:374] T d35fe4ff3d474b66b06a2bef87dfbfdd P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:56.876502 8417 raft_consensus.cc:729] T d35fe4ff3d474b66b06a2bef87dfbfdd P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee5ade571f0349a1b4e3fa4edbeae2e5, State: Initialized, Role: FOLLOWER
I20260928 02:34:56.876602 8417 consensus_queue.cc:260] T d35fe4ff3d474b66b06a2bef87dfbfdd P ee5ade571f0349a1b4e3fa4edbeae2e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } }
I20260928 02:34:56.876592 8414 raft_consensus.cc:348] T d35fe4ff3d474b66b06a2bef87dfbfdd P a1187d9ed1b24dac9b4887d60838afb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } }
I20260928 02:34:56.876703 8414 raft_consensus.cc:374] T d35fe4ff3d474b66b06a2bef87dfbfdd P a1187d9ed1b24dac9b4887d60838afb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:56.876748 8414 raft_consensus.cc:729] T d35fe4ff3d474b66b06a2bef87dfbfdd P a1187d9ed1b24dac9b4887d60838afb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1187d9ed1b24dac9b4887d60838afb9, State: Initialized, Role: FOLLOWER
I20260928 02:34:56.876850 8417 ts_tablet_manager.cc:1428] T d35fe4ff3d474b66b06a2bef87dfbfdd P ee5ade571f0349a1b4e3fa4edbeae2e5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 02:34:56.876878 8248 heartbeater.cc:503] Master 127.0.98.62:35019 was elected leader, sending a full tablet report...
I20260928 02:34:56.876884 8414 consensus_queue.cc:260] T d35fe4ff3d474b66b06a2bef87dfbfdd P a1187d9ed1b24dac9b4887d60838afb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } }
I20260928 02:34:56.877120 8414 ts_tablet_manager.cc:1428] T d35fe4ff3d474b66b06a2bef87dfbfdd P a1187d9ed1b24dac9b4887d60838afb9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:34:56.877156 8388 heartbeater.cc:503] Master 127.0.98.62:35019 was elected leader, sending a full tablet report...
W20260928 02:34:56.918175 8249 tablet.cc:2369] T d35fe4ff3d474b66b06a2bef87dfbfdd P ee5ade571f0349a1b4e3fa4edbeae2e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:34:57.014550 8109 tablet.cc:2369] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:34:57.070262 8389 tablet.cc:2369] T d35fe4ff3d474b66b06a2bef87dfbfdd P a1187d9ed1b24dac9b4887d60838afb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:34:57.126047 8419 raft_consensus.cc:482] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:34:57.126247 8419 raft_consensus.cc:504] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } }
I20260928 02:34:57.126623 8419 leader_election.cc:288] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1187d9ed1b24dac9b4887d60838afb9 (127.0.98.3:35633), ee5ade571f0349a1b4e3fa4edbeae2e5 (127.0.98.2:42811)
I20260928 02:34:57.127732 8421 raft_consensus.cc:482] T d35fe4ff3d474b66b06a2bef87dfbfdd P a1187d9ed1b24dac9b4887d60838afb9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:34:57.127839 8421 raft_consensus.cc:504] T d35fe4ff3d474b66b06a2bef87dfbfdd P a1187d9ed1b24dac9b4887d60838afb9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } }
I20260928 02:34:57.128168 8421 leader_election.cc:288] T d35fe4ff3d474b66b06a2bef87dfbfdd P a1187d9ed1b24dac9b4887d60838afb9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee5ade571f0349a1b4e3fa4edbeae2e5 (127.0.98.2:42811), 785d3ca35dcd4aea838abdfdf3584460 (127.0.98.1:43567)
I20260928 02:34:57.130023 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:57.126882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41017 (local address 127.0.98.3:35633)
0928 02:34:57.126937 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:57.126939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:57.127040 (+ 101us) server_negotiation.cc:407] Connection header received
0928 02:34:57.127096 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:57.127097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:57.127106 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:57.127147 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:57.127276 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.127807 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.129135 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.129693 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.129731 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:57.129740 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:57.129929 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 02:34:57.129965 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 02:34:57.130034 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:57.126814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35633 (local address 127.0.98.1:41017)
0928 02:34:57.126969 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:57.126978 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:57.127037 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:57.127148 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:57.127149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:57.127153 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:57.127218 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:57.127220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.127894 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.127895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:57.129059 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:57.129060 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.129801 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.129801 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:57.129865 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:57.129868 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:34:57.129912 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 02:34:57.129948 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":36,"thread_start_us":76,"threads_started":1}
I20260928 02:34:57.130242 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:57.126871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38427 (local address 127.0.98.2:42811)
0928 02:34:57.126951 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:57.126953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:57.126989 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:34:57.127074 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:57.127075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:57.127087 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:57.127137 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:57.127268 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.127786 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.129199 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.129738 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.129774 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:57.129783 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:57.130127 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 02:34:57.130181 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:34:57.130415 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:57.126813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42811 (local address 127.0.98.1:38427)
0928 02:34:57.126928 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:57.126939 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:57.127004 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:57.127147 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:57.127148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:57.127151 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:57.127211 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:57.127213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.127874 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.127875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:57.129104 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:57.129106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.129986 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.129987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:57.130051 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:57.130054 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:34:57.130308 (+ 254us) client_negotiation.cc:240] Negotiation successful
0928 02:34:57.130346 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":52,"thread_start_us":65,"threads_started":1}
I20260928 02:34:57.130384 8334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d35fe4ff3d474b66b06a2bef87dfbfdd" candidate_uuid: "785d3ca35dcd4aea838abdfdf3584460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1187d9ed1b24dac9b4887d60838afb9" is_pre_election: true
I20260928 02:34:57.130472 8334 raft_consensus.cc:2454] T d35fe4ff3d474b66b06a2bef87dfbfdd P a1187d9ed1b24dac9b4887d60838afb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 785d3ca35dcd4aea838abdfdf3584460 in term 0.
I20260928 02:34:57.130653 7993 leader_election.cc:302] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 785d3ca35dcd4aea838abdfdf3584460, a1187d9ed1b24dac9b4887d60838afb9; no voters:
I20260928 02:34:57.130772 8419 raft_consensus.cc:2781] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:34:57.130821 8419 raft_consensus.cc:482] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:34:57.130842 8419 raft_consensus.cc:3037] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:57.131036 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d35fe4ff3d474b66b06a2bef87dfbfdd" candidate_uuid: "785d3ca35dcd4aea838abdfdf3584460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" is_pre_election: true
I20260928 02:34:57.131183 8202 raft_consensus.cc:2454] T d35fe4ff3d474b66b06a2bef87dfbfdd P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 785d3ca35dcd4aea838abdfdf3584460 in term 0.
I20260928 02:34:57.131690 8419 raft_consensus.cc:504] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } }
I20260928 02:34:57.131811 8419 leader_election.cc:288] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460 [CANDIDATE]: Term 1 election: Requested vote from peers a1187d9ed1b24dac9b4887d60838afb9 (127.0.98.3:35633), ee5ade571f0349a1b4e3fa4edbeae2e5 (127.0.98.2:42811)
I20260928 02:34:57.131816 8283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:57.128296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42811 (local address 127.0.98.3:39847)
0928 02:34:57.128404 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:57.128415 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:57.128455 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:57.128802 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:57.128803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:57.128805 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:57.128864 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:57.128865 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.129774 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.129774 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:57.130779 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:57.130781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.131578 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.131579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:57.131650 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:57.131653 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:34:57.131703 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 02:34:57.131742 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":21}
I20260928 02:34:57.132004 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:57.128520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39847 (local address 127.0.98.2:42811)
0928 02:34:57.128674 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:57.128675 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:57.128684 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:34:57.128702 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:57.128702 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:57.128710 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:57.128926 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:57.128953 (+ 27us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.129685 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.130886 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.131500 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.131571 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:57.131582 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:57.131891 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 02:34:57.131930 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
I20260928 02:34:57.132083 8334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d35fe4ff3d474b66b06a2bef87dfbfdd" candidate_uuid: "785d3ca35dcd4aea838abdfdf3584460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1187d9ed1b24dac9b4887d60838afb9"
I20260928 02:34:57.132150 8334 raft_consensus.cc:3037] T d35fe4ff3d474b66b06a2bef87dfbfdd P a1187d9ed1b24dac9b4887d60838afb9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:57.132148 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d35fe4ff3d474b66b06a2bef87dfbfdd" candidate_uuid: "a1187d9ed1b24dac9b4887d60838afb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" is_pre_election: true
I20260928 02:34:57.132217 8201 raft_consensus.cc:2454] T d35fe4ff3d474b66b06a2bef87dfbfdd P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1187d9ed1b24dac9b4887d60838afb9 in term 0.
I20260928 02:34:57.132368 8274 leader_election.cc:302] T d35fe4ff3d474b66b06a2bef87dfbfdd P a1187d9ed1b24dac9b4887d60838afb9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1187d9ed1b24dac9b4887d60838afb9, ee5ade571f0349a1b4e3fa4edbeae2e5; no voters:
I20260928 02:34:57.132905 8334 raft_consensus.cc:2454] T d35fe4ff3d474b66b06a2bef87dfbfdd P a1187d9ed1b24dac9b4887d60838afb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 785d3ca35dcd4aea838abdfdf3584460 in term 1.
I20260928 02:34:57.132984 8421 raft_consensus.cc:2741] T d35fe4ff3d474b66b06a2bef87dfbfdd P a1187d9ed1b24dac9b4887d60838afb9 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260928 02:34:57.133049 7993 leader_election.cc:302] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 785d3ca35dcd4aea838abdfdf3584460, a1187d9ed1b24dac9b4887d60838afb9; no voters:
I20260928 02:34:57.133087 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d35fe4ff3d474b66b06a2bef87dfbfdd" candidate_uuid: "785d3ca35dcd4aea838abdfdf3584460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5"
I20260928 02:34:57.133126 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:57.128369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45013 (local address 127.0.98.1:43567)
0928 02:34:57.128636 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:57.128638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:57.128730 (+ 92us) server_negotiation.cc:407] Connection header received
0928 02:34:57.128804 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:57.128805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:57.128813 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:57.128838 (+ 25us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:34:57.129003 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.129685 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.132462 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.132808 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.132862 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:57.132869 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:57.133035 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 02:34:57.133070 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":251}
I20260928 02:34:57.133153 8202 raft_consensus.cc:3037] T d35fe4ff3d474b66b06a2bef87dfbfdd P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:57.133211 8419 raft_consensus.cc:2781] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:57.133325 8419 raft_consensus.cc:686] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460 [term 1 LEADER]: Becoming Leader. State: Replica: 785d3ca35dcd4aea838abdfdf3584460, State: Running, Role: LEADER
I20260928 02:34:57.133351 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:57.128296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43567 (local address 127.0.98.3:45013)
0928 02:34:57.128684 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:57.128692 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:57.128747 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:57.128885 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:57.128886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:57.128887 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:34:57.128946 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:57.128948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.129772 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.129772 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:57.132397 (+ 2625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:57.132399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.132892 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.132893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:57.132956 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:57.132959 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:34:57.133202 (+ 243us) client_negotiation.cc:240] Negotiation successful
0928 02:34:57.133240 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":63,"thread_start_us":84,"threads_started":1}
I20260928 02:34:57.133419 8419 consensus_queue.cc:237] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } }
I20260928 02:34:57.133582 8062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d35fe4ff3d474b66b06a2bef87dfbfdd" candidate_uuid: "a1187d9ed1b24dac9b4887d60838afb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "785d3ca35dcd4aea838abdfdf3584460" is_pre_election: true
I20260928 02:34:57.134274 8202 raft_consensus.cc:2454] T d35fe4ff3d474b66b06a2bef87dfbfdd P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 785d3ca35dcd4aea838abdfdf3584460 in term 1.
I20260928 02:34:57.134207 7905 catalog_manager.cc:5495] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460 reported cstate change: term changed from 0 to 1, leader changed from <none> to 785d3ca35dcd4aea838abdfdf3584460 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "785d3ca35dcd4aea838abdfdf3584460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } health_report { overall_health: HEALTHY } } }
I20260928 02:34:57.151432 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:57.149449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47692 (local address 127.0.98.1:43567)
0928 02:34:57.149485 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:57.149487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:57.149496 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:34:57.149553 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:57.149554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:57.149568 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:57.149626 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:34:57.149718 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.150159 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.150819 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.151009 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.151062 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:57.151158 (+ 96us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:34:57.151279 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:34:57.151352 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:57.151366 (+ 14us) server_negotiation.cc:299] Negotiation successful
0928 02:34:57.151381 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:34:57.153054 8334 raft_consensus.cc:1261] T d35fe4ff3d474b66b06a2bef87dfbfdd P a1187d9ed1b24dac9b4887d60838afb9 [term 1 FOLLOWER]: Refusing update from remote peer 785d3ca35dcd4aea838abdfdf3584460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:57.153115 8202 raft_consensus.cc:1261] T d35fe4ff3d474b66b06a2bef87dfbfdd P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 1 FOLLOWER]: Refusing update from remote peer 785d3ca35dcd4aea838abdfdf3584460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:57.153381 8426 consensus_queue.cc:1035] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:57.153483 8419 consensus_queue.cc:1035] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:57.155539 8429 mvcc.cc:205] Tried to move back new op lower bound from 7334145626736644096 to 7334145626659037184. Current Snapshot: MvccSnapshot[applied={T|T < 7334145626736644096}]
I20260928 02:34:57.157509 7905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:34:57.157586 7905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:34:57.158228 7905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca69e07373f44b6096c5c7cd6b96bdd5: Sending DeleteTablet for 3 replicas of tablet d35fe4ff3d474b66b06a2bef87dfbfdd
I20260928 02:34:57.158545 8322 tablet_service.cc:1514] Processing DeleteTablet for tablet d35fe4ff3d474b66b06a2bef87dfbfdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32856
I20260928 02:34:57.158632 8182 tablet_service.cc:1514] Processing DeleteTablet for tablet d35fe4ff3d474b66b06a2bef87dfbfdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40210
I20260928 02:34:57.158756 8042 tablet_service.cc:1514] Processing DeleteTablet for tablet d35fe4ff3d474b66b06a2bef87dfbfdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47682
I20260928 02:34:57.158870 8440 tablet_replica.cc:330] T d35fe4ff3d474b66b06a2bef87dfbfdd P ee5ade571f0349a1b4e3fa4edbeae2e5: stopping tablet replica
I20260928 02:34:57.158926 8441 tablet_replica.cc:330] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460: stopping tablet replica
I20260928 02:34:57.158911 7905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:34:57.159010 8441 raft_consensus.cc:2229] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460 [term 1 LEADER]: Raft consensus shutting down.
W20260928 02:34:57.159040 7905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:34:57.159142 8441 raft_consensus.cc:2258] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:57.159235 8440 raft_consensus.cc:2229] T d35fe4ff3d474b66b06a2bef87dfbfdd P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:34:57.159375 8440 raft_consensus.cc:2258] T d35fe4ff3d474b66b06a2bef87dfbfdd P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:57.159590 8441 ts_tablet_manager.cc:1905] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:57.160352 8042 tablet_service.cc:1467] Processing CreateTablet for tablet 806b1f5f2d2c421893acbffcaafdc9a9 (DEFAULT_TABLE table=test-table [id=bdce2262a3bb4a84a111d97d2a3dc31c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:57.160377 8322 tablet_service.cc:1467] Processing CreateTablet for tablet 806b1f5f2d2c421893acbffcaafdc9a9 (DEFAULT_TABLE table=test-table [id=bdce2262a3bb4a84a111d97d2a3dc31c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:57.160455 8322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 806b1f5f2d2c421893acbffcaafdc9a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:57.160455 8042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 806b1f5f2d2c421893acbffcaafdc9a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:57.160915 8182 tablet_service.cc:1467] Processing CreateTablet for tablet 806b1f5f2d2c421893acbffcaafdc9a9 (DEFAULT_TABLE table=test-table [id=bdce2262a3bb4a84a111d97d2a3dc31c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:57.161018 8182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 806b1f5f2d2c421893acbffcaafdc9a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:57.161301 8440 ts_tablet_manager.cc:1905] T d35fe4ff3d474b66b06a2bef87dfbfdd P ee5ade571f0349a1b4e3fa4edbeae2e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:57.161401 8439 tablet_replica.cc:330] T d35fe4ff3d474b66b06a2bef87dfbfdd P a1187d9ed1b24dac9b4887d60838afb9: stopping tablet replica
I20260928 02:34:57.161520 8439 raft_consensus.cc:2229] T d35fe4ff3d474b66b06a2bef87dfbfdd P a1187d9ed1b24dac9b4887d60838afb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:34:57.161691 8414 tablet_bootstrap.cc:492] T 806b1f5f2d2c421893acbffcaafdc9a9 P a1187d9ed1b24dac9b4887d60838afb9: Bootstrap starting.
I20260928 02:34:57.162089 8414 tablet_bootstrap.cc:654] T 806b1f5f2d2c421893acbffcaafdc9a9 P a1187d9ed1b24dac9b4887d60838afb9: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:57.162319 8417 tablet_bootstrap.cc:492] T 806b1f5f2d2c421893acbffcaafdc9a9 P ee5ade571f0349a1b4e3fa4edbeae2e5: Bootstrap starting.
I20260928 02:34:57.161656 8439 raft_consensus.cc:2258] T d35fe4ff3d474b66b06a2bef87dfbfdd P a1187d9ed1b24dac9b4887d60838afb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:57.162693 8417 tablet_bootstrap.cc:654] T 806b1f5f2d2c421893acbffcaafdc9a9 P ee5ade571f0349a1b4e3fa4edbeae2e5: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:57.162716 8441 ts_tablet_manager.cc:1918] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:34:57.162775 8441 log.cc:1196] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-0/wal/wals/d35fe4ff3d474b66b06a2bef87dfbfdd
I20260928 02:34:57.162933 8440 ts_tablet_manager.cc:1918] T d35fe4ff3d474b66b06a2bef87dfbfdd P ee5ade571f0349a1b4e3fa4edbeae2e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:34:57.163012 8441 ts_tablet_manager.cc:1939] T d35fe4ff3d474b66b06a2bef87dfbfdd P 785d3ca35dcd4aea838abdfdf3584460: Deleting consensus metadata
I20260928 02:34:57.163060 8440 log.cc:1196] T d35fe4ff3d474b66b06a2bef87dfbfdd P ee5ade571f0349a1b4e3fa4edbeae2e5: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-1/wal/wals/d35fe4ff3d474b66b06a2bef87dfbfdd
I20260928 02:34:57.163283 8440 ts_tablet_manager.cc:1939] T d35fe4ff3d474b66b06a2bef87dfbfdd P ee5ade571f0349a1b4e3fa4edbeae2e5: Deleting consensus metadata
I20260928 02:34:57.163357 8417 tablet_bootstrap.cc:492] T 806b1f5f2d2c421893acbffcaafdc9a9 P ee5ade571f0349a1b4e3fa4edbeae2e5: No bootstrap required, opened a new log
I20260928 02:34:57.163379 8439 ts_tablet_manager.cc:1905] T d35fe4ff3d474b66b06a2bef87dfbfdd P a1187d9ed1b24dac9b4887d60838afb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:57.163398 8417 ts_tablet_manager.cc:1397] T 806b1f5f2d2c421893acbffcaafdc9a9 P ee5ade571f0349a1b4e3fa4edbeae2e5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 02:34:57.163527 8417 raft_consensus.cc:348] T 806b1f5f2d2c421893acbffcaafdc9a9 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.163582 8417 raft_consensus.cc:374] T 806b1f5f2d2c421893acbffcaafdc9a9 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:57.163600 8417 raft_consensus.cc:729] T 806b1f5f2d2c421893acbffcaafdc9a9 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee5ade571f0349a1b4e3fa4edbeae2e5, State: Initialized, Role: FOLLOWER
I20260928 02:34:57.163625 7890 catalog_manager.cc:4841] TS ee5ade571f0349a1b4e3fa4edbeae2e5 (127.0.98.2:42811): tablet d35fe4ff3d474b66b06a2bef87dfbfdd (table test-table [id=47ed87d2157545f0beb4d4e4f2d86163]) successfully deleted
I20260928 02:34:57.163640 8417 consensus_queue.cc:260] T 806b1f5f2d2c421893acbffcaafdc9a9 P ee5ade571f0349a1b4e3fa4edbeae2e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.163713 8417 ts_tablet_manager.cc:1428] T 806b1f5f2d2c421893acbffcaafdc9a9 P ee5ade571f0349a1b4e3fa4edbeae2e5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:57.164004 8413 tablet_bootstrap.cc:492] T 806b1f5f2d2c421893acbffcaafdc9a9 P 785d3ca35dcd4aea838abdfdf3584460: Bootstrap starting.
I20260928 02:34:57.164358 8413 tablet_bootstrap.cc:654] T 806b1f5f2d2c421893acbffcaafdc9a9 P 785d3ca35dcd4aea838abdfdf3584460: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:57.164695 8439 ts_tablet_manager.cc:1918] T d35fe4ff3d474b66b06a2bef87dfbfdd P a1187d9ed1b24dac9b4887d60838afb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:34:57.164768 8439 log.cc:1196] T d35fe4ff3d474b66b06a2bef87dfbfdd P a1187d9ed1b24dac9b4887d60838afb9: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-2/wal/wals/d35fe4ff3d474b66b06a2bef87dfbfdd
I20260928 02:34:57.164891 8413 tablet_bootstrap.cc:492] T 806b1f5f2d2c421893acbffcaafdc9a9 P 785d3ca35dcd4aea838abdfdf3584460: No bootstrap required, opened a new log
I20260928 02:34:57.164939 8413 ts_tablet_manager.cc:1397] T 806b1f5f2d2c421893acbffcaafdc9a9 P 785d3ca35dcd4aea838abdfdf3584460: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:34:57.164968 8439 ts_tablet_manager.cc:1939] T d35fe4ff3d474b66b06a2bef87dfbfdd P a1187d9ed1b24dac9b4887d60838afb9: Deleting consensus metadata
I20260928 02:34:57.165067 8413 raft_consensus.cc:348] T 806b1f5f2d2c421893acbffcaafdc9a9 P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.165132 8413 raft_consensus.cc:374] T 806b1f5f2d2c421893acbffcaafdc9a9 P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:57.165158 8413 raft_consensus.cc:729] T 806b1f5f2d2c421893acbffcaafdc9a9 P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 785d3ca35dcd4aea838abdfdf3584460, State: Initialized, Role: FOLLOWER
I20260928 02:34:57.165195 8413 consensus_queue.cc:260] T 806b1f5f2d2c421893acbffcaafdc9a9 P 785d3ca35dcd4aea838abdfdf3584460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.165294 8413 ts_tablet_manager.cc:1428] T 806b1f5f2d2c421893acbffcaafdc9a9 P 785d3ca35dcd4aea838abdfdf3584460: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:57.165428 7889 catalog_manager.cc:4841] TS 785d3ca35dcd4aea838abdfdf3584460 (127.0.98.1:43567): tablet d35fe4ff3d474b66b06a2bef87dfbfdd (table test-table [id=47ed87d2157545f0beb4d4e4f2d86163]) successfully deleted
I20260928 02:34:57.165627 7889 catalog_manager.cc:4841] TS a1187d9ed1b24dac9b4887d60838afb9 (127.0.98.3:35633): tablet d35fe4ff3d474b66b06a2bef87dfbfdd (table test-table [id=47ed87d2157545f0beb4d4e4f2d86163]) successfully deleted
I20260928 02:34:57.173082 8414 tablet_bootstrap.cc:492] T 806b1f5f2d2c421893acbffcaafdc9a9 P a1187d9ed1b24dac9b4887d60838afb9: No bootstrap required, opened a new log
I20260928 02:34:57.173139 8414 ts_tablet_manager.cc:1397] T 806b1f5f2d2c421893acbffcaafdc9a9 P a1187d9ed1b24dac9b4887d60838afb9: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260928 02:34:57.173296 8414 raft_consensus.cc:348] T 806b1f5f2d2c421893acbffcaafdc9a9 P a1187d9ed1b24dac9b4887d60838afb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.173364 8414 raft_consensus.cc:374] T 806b1f5f2d2c421893acbffcaafdc9a9 P a1187d9ed1b24dac9b4887d60838afb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:57.173389 8414 raft_consensus.cc:729] T 806b1f5f2d2c421893acbffcaafdc9a9 P a1187d9ed1b24dac9b4887d60838afb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1187d9ed1b24dac9b4887d60838afb9, State: Initialized, Role: FOLLOWER
I20260928 02:34:57.173437 8414 consensus_queue.cc:260] T 806b1f5f2d2c421893acbffcaafdc9a9 P a1187d9ed1b24dac9b4887d60838afb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.173525 8414 ts_tablet_manager.cc:1428] T 806b1f5f2d2c421893acbffcaafdc9a9 P a1187d9ed1b24dac9b4887d60838afb9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:57.178640 8434 raft_consensus.cc:482] T 806b1f5f2d2c421893acbffcaafdc9a9 P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:34:57.178697 8434 raft_consensus.cc:504] T 806b1f5f2d2c421893acbffcaafdc9a9 P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.178833 8434 leader_election.cc:288] T 806b1f5f2d2c421893acbffcaafdc9a9 P 785d3ca35dcd4aea838abdfdf3584460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee5ade571f0349a1b4e3fa4edbeae2e5 (127.0.98.2:42811), a1187d9ed1b24dac9b4887d60838afb9 (127.0.98.3:35633)
I20260928 02:34:57.179009 8334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "806b1f5f2d2c421893acbffcaafdc9a9" candidate_uuid: "785d3ca35dcd4aea838abdfdf3584460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1187d9ed1b24dac9b4887d60838afb9" is_pre_election: true
I20260928 02:34:57.179009 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "806b1f5f2d2c421893acbffcaafdc9a9" candidate_uuid: "785d3ca35dcd4aea838abdfdf3584460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" is_pre_election: true
I20260928 02:34:57.179083 8334 raft_consensus.cc:2454] T 806b1f5f2d2c421893acbffcaafdc9a9 P a1187d9ed1b24dac9b4887d60838afb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 785d3ca35dcd4aea838abdfdf3584460 in term 0.
I20260928 02:34:57.179083 8202 raft_consensus.cc:2454] T 806b1f5f2d2c421893acbffcaafdc9a9 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 785d3ca35dcd4aea838abdfdf3584460 in term 0.
I20260928 02:34:57.179226 7994 leader_election.cc:302] T 806b1f5f2d2c421893acbffcaafdc9a9 P 785d3ca35dcd4aea838abdfdf3584460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 785d3ca35dcd4aea838abdfdf3584460, ee5ade571f0349a1b4e3fa4edbeae2e5; no voters:
I20260928 02:34:57.179366 8434 raft_consensus.cc:2781] T 806b1f5f2d2c421893acbffcaafdc9a9 P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:34:57.179399 8434 raft_consensus.cc:482] T 806b1f5f2d2c421893acbffcaafdc9a9 P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:34:57.179410 8434 raft_consensus.cc:3037] T 806b1f5f2d2c421893acbffcaafdc9a9 P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:57.179975 8434 raft_consensus.cc:504] T 806b1f5f2d2c421893acbffcaafdc9a9 P 785d3ca35dcd4aea838abdfdf3584460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.180073 8434 leader_election.cc:288] T 806b1f5f2d2c421893acbffcaafdc9a9 P 785d3ca35dcd4aea838abdfdf3584460 [CANDIDATE]: Term 1 election: Requested vote from peers ee5ade571f0349a1b4e3fa4edbeae2e5 (127.0.98.2:42811), a1187d9ed1b24dac9b4887d60838afb9 (127.0.98.3:35633)
I20260928 02:34:57.180191 8334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "806b1f5f2d2c421893acbffcaafdc9a9" candidate_uuid: "785d3ca35dcd4aea838abdfdf3584460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1187d9ed1b24dac9b4887d60838afb9"
I20260928 02:34:57.180254 8334 raft_consensus.cc:3037] T 806b1f5f2d2c421893acbffcaafdc9a9 P a1187d9ed1b24dac9b4887d60838afb9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:57.180239 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "806b1f5f2d2c421893acbffcaafdc9a9" candidate_uuid: "785d3ca35dcd4aea838abdfdf3584460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5"
I20260928 02:34:57.180305 8202 raft_consensus.cc:3037] T 806b1f5f2d2c421893acbffcaafdc9a9 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:57.180757 8334 raft_consensus.cc:2454] T 806b1f5f2d2c421893acbffcaafdc9a9 P a1187d9ed1b24dac9b4887d60838afb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 785d3ca35dcd4aea838abdfdf3584460 in term 1.
I20260928 02:34:57.180898 7993 leader_election.cc:302] T 806b1f5f2d2c421893acbffcaafdc9a9 P 785d3ca35dcd4aea838abdfdf3584460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 785d3ca35dcd4aea838abdfdf3584460, a1187d9ed1b24dac9b4887d60838afb9; no voters:
I20260928 02:34:57.180967 8434 raft_consensus.cc:2781] T 806b1f5f2d2c421893acbffcaafdc9a9 P 785d3ca35dcd4aea838abdfdf3584460 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:57.180984 8202 raft_consensus.cc:2454] T 806b1f5f2d2c421893acbffcaafdc9a9 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 785d3ca35dcd4aea838abdfdf3584460 in term 1.
I20260928 02:34:57.181005 8434 raft_consensus.cc:686] T 806b1f5f2d2c421893acbffcaafdc9a9 P 785d3ca35dcd4aea838abdfdf3584460 [term 1 LEADER]: Becoming Leader. State: Replica: 785d3ca35dcd4aea838abdfdf3584460, State: Running, Role: LEADER
I20260928 02:34:57.181041 8434 consensus_queue.cc:237] T 806b1f5f2d2c421893acbffcaafdc9a9 P 785d3ca35dcd4aea838abdfdf3584460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.181468 7905 catalog_manager.cc:5495] T 806b1f5f2d2c421893acbffcaafdc9a9 P 785d3ca35dcd4aea838abdfdf3584460 reported cstate change: term changed from 0 to 1, leader changed from <none> to 785d3ca35dcd4aea838abdfdf3584460 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "785d3ca35dcd4aea838abdfdf3584460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } health_report { overall_health: UNKNOWN } } }
Sep 28 02:34:57 dist-test-slave-fz47 krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562896, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 02:34:57.211550 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:57.201895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35019 (local address 127.0.0.1:38292)
0928 02:34:57.202041 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:57.202049 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:57.202111 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:57.202222 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:57.202223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:57.202528 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:34:57.202573 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:57.202575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.203196 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.203197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:57.203714 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:57.203715 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.204009 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.204010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:57.204054 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:57.207765 (+ 3711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:34:57.207769 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:34:57.209238 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:34:57.210608 (+ 1370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:57.210614 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:57.210617 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:57.210805 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:57.210960 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:34:57.210960 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:34:57.210961 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:34:57.211012 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:34:57.211206 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:34:57.211213 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:34:57.211257 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 02:34:57.211316 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 02:34:57.211434 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":4,"thread_start_us":96,"threads_started":1}
I20260928 02:34:57.211552 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:57.201930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38292 (local address 127.0.98.62:35019)
0928 02:34:57.201978 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:57.201980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:57.202106 (+ 126us) server_negotiation.cc:407] Connection header received
0928 02:34:57.202159 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:57.202160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:57.202169 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:57.202214 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:57.202651 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.203131 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.203788 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.203955 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.204007 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:57.209337 (+ 5330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:57.209339 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:57.209339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:57.209346 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:57.210542 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:57.210864 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:57.210865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:57.210866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:57.210910 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:57.211062 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:57.211062 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:57.211063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:57.211157 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:57.211209 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:57.211341 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 02:34:57.211477 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":28}
I20260928 02:34:57.213431 7902 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38292:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:34:57.213606 7902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:34:57.214843 8182 tablet_service.cc:1467] Processing CreateTablet for tablet 1b9e98043c5a470c891831081e3ce4f1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=db6e094632954478bdc717cc04cfaea2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:34:57.214867 8322 tablet_service.cc:1467] Processing CreateTablet for tablet 1b9e98043c5a470c891831081e3ce4f1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=db6e094632954478bdc717cc04cfaea2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:34:57.214867 8042 tablet_service.cc:1467] Processing CreateTablet for tablet 1b9e98043c5a470c891831081e3ce4f1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=db6e094632954478bdc717cc04cfaea2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:34:57.215001 8182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b9e98043c5a470c891831081e3ce4f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:57.215001 8042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b9e98043c5a470c891831081e3ce4f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:57.215001 8322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b9e98043c5a470c891831081e3ce4f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:57.216576 8414 tablet_bootstrap.cc:492] T 1b9e98043c5a470c891831081e3ce4f1 P a1187d9ed1b24dac9b4887d60838afb9: Bootstrap starting.
I20260928 02:34:57.216995 8414 tablet_bootstrap.cc:654] T 1b9e98043c5a470c891831081e3ce4f1 P a1187d9ed1b24dac9b4887d60838afb9: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:57.217342 8413 tablet_bootstrap.cc:492] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460: Bootstrap starting.
I20260928 02:34:57.217577 8414 tablet_bootstrap.cc:492] T 1b9e98043c5a470c891831081e3ce4f1 P a1187d9ed1b24dac9b4887d60838afb9: No bootstrap required, opened a new log
I20260928 02:34:57.217612 8414 ts_tablet_manager.cc:1397] T 1b9e98043c5a470c891831081e3ce4f1 P a1187d9ed1b24dac9b4887d60838afb9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 02:34:57.217694 8413 tablet_bootstrap.cc:654] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:57.217721 8414 raft_consensus.cc:348] T 1b9e98043c5a470c891831081e3ce4f1 P a1187d9ed1b24dac9b4887d60838afb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.217764 8414 raft_consensus.cc:374] T 1b9e98043c5a470c891831081e3ce4f1 P a1187d9ed1b24dac9b4887d60838afb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:57.217777 8414 raft_consensus.cc:729] T 1b9e98043c5a470c891831081e3ce4f1 P a1187d9ed1b24dac9b4887d60838afb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1187d9ed1b24dac9b4887d60838afb9, State: Initialized, Role: FOLLOWER
I20260928 02:34:57.217813 8414 consensus_queue.cc:260] T 1b9e98043c5a470c891831081e3ce4f1 P a1187d9ed1b24dac9b4887d60838afb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.217895 8414 ts_tablet_manager.cc:1428] T 1b9e98043c5a470c891831081e3ce4f1 P a1187d9ed1b24dac9b4887d60838afb9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:57.217931 8421 tablet_replica.cc:439] T 1b9e98043c5a470c891831081e3ce4f1 P a1187d9ed1b24dac9b4887d60838afb9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } } }
I20260928 02:34:57.217993 8421 tablet_replica.cc:442] T 1b9e98043c5a470c891831081e3ce4f1 P a1187d9ed1b24dac9b4887d60838afb9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:34:57.218005 8417 tablet_bootstrap.cc:492] T 1b9e98043c5a470c891831081e3ce4f1 P ee5ade571f0349a1b4e3fa4edbeae2e5: Bootstrap starting.
I20260928 02:34:57.218354 8417 tablet_bootstrap.cc:654] T 1b9e98043c5a470c891831081e3ce4f1 P ee5ade571f0349a1b4e3fa4edbeae2e5: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:57.218895 8417 tablet_bootstrap.cc:492] T 1b9e98043c5a470c891831081e3ce4f1 P ee5ade571f0349a1b4e3fa4edbeae2e5: No bootstrap required, opened a new log
I20260928 02:34:57.218935 8417 ts_tablet_manager.cc:1397] T 1b9e98043c5a470c891831081e3ce4f1 P ee5ade571f0349a1b4e3fa4edbeae2e5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 02:34:57.219074 8417 raft_consensus.cc:348] T 1b9e98043c5a470c891831081e3ce4f1 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.219138 8417 raft_consensus.cc:374] T 1b9e98043c5a470c891831081e3ce4f1 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:57.219156 8417 raft_consensus.cc:729] T 1b9e98043c5a470c891831081e3ce4f1 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee5ade571f0349a1b4e3fa4edbeae2e5, State: Initialized, Role: FOLLOWER
I20260928 02:34:57.219193 8417 consensus_queue.cc:260] T 1b9e98043c5a470c891831081e3ce4f1 P ee5ade571f0349a1b4e3fa4edbeae2e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.219269 8417 ts_tablet_manager.cc:1428] T 1b9e98043c5a470c891831081e3ce4f1 P ee5ade571f0349a1b4e3fa4edbeae2e5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:57.219267 8413 tablet_bootstrap.cc:492] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460: No bootstrap required, opened a new log
I20260928 02:34:57.219308 8413 ts_tablet_manager.cc:1397] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:34:57.219313 8420 tablet_replica.cc:439] T 1b9e98043c5a470c891831081e3ce4f1 P ee5ade571f0349a1b4e3fa4edbeae2e5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } } }
I20260928 02:34:57.219370 8420 tablet_replica.cc:442] T 1b9e98043c5a470c891831081e3ce4f1 P ee5ade571f0349a1b4e3fa4edbeae2e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:34:57.219398 8413 raft_consensus.cc:348] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.219453 8413 raft_consensus.cc:374] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:57.219472 8413 raft_consensus.cc:729] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 785d3ca35dcd4aea838abdfdf3584460, State: Initialized, Role: FOLLOWER
I20260928 02:34:57.219514 8413 consensus_queue.cc:260] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.219587 8413 ts_tablet_manager.cc:1428] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:57.219609 8426 tablet_replica.cc:439] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } } }
I20260928 02:34:57.219688 8426 tablet_replica.cc:442] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:34:57.309659 8426 raft_consensus.cc:482] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:34:57.309751 8426 raft_consensus.cc:504] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.309886 8426 leader_election.cc:288] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee5ade571f0349a1b4e3fa4edbeae2e5 (127.0.98.2:42811), a1187d9ed1b24dac9b4887d60838afb9 (127.0.98.3:35633)
I20260928 02:34:57.310104 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9e98043c5a470c891831081e3ce4f1" candidate_uuid: "785d3ca35dcd4aea838abdfdf3584460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" is_pre_election: true
I20260928 02:34:57.310103 8334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9e98043c5a470c891831081e3ce4f1" candidate_uuid: "785d3ca35dcd4aea838abdfdf3584460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1187d9ed1b24dac9b4887d60838afb9" is_pre_election: true
I20260928 02:34:57.310185 8334 raft_consensus.cc:2454] T 1b9e98043c5a470c891831081e3ce4f1 P a1187d9ed1b24dac9b4887d60838afb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 785d3ca35dcd4aea838abdfdf3584460 in term 0.
I20260928 02:34:57.310186 8202 raft_consensus.cc:2454] T 1b9e98043c5a470c891831081e3ce4f1 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 785d3ca35dcd4aea838abdfdf3584460 in term 0.
I20260928 02:34:57.310343 7994 leader_election.cc:302] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 785d3ca35dcd4aea838abdfdf3584460, ee5ade571f0349a1b4e3fa4edbeae2e5; no voters:
I20260928 02:34:57.310453 8426 raft_consensus.cc:2781] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:34:57.310501 8426 raft_consensus.cc:482] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:34:57.310519 8426 raft_consensus.cc:3037] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:57.311175 8426 raft_consensus.cc:504] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.311292 8426 leader_election.cc:288] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460 [CANDIDATE]: Term 1 election: Requested vote from peers ee5ade571f0349a1b4e3fa4edbeae2e5 (127.0.98.2:42811), a1187d9ed1b24dac9b4887d60838afb9 (127.0.98.3:35633)
I20260928 02:34:57.311411 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9e98043c5a470c891831081e3ce4f1" candidate_uuid: "785d3ca35dcd4aea838abdfdf3584460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5"
I20260928 02:34:57.311429 8334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9e98043c5a470c891831081e3ce4f1" candidate_uuid: "785d3ca35dcd4aea838abdfdf3584460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1187d9ed1b24dac9b4887d60838afb9"
I20260928 02:34:57.311467 8202 raft_consensus.cc:3037] T 1b9e98043c5a470c891831081e3ce4f1 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:57.311486 8334 raft_consensus.cc:3037] T 1b9e98043c5a470c891831081e3ce4f1 P a1187d9ed1b24dac9b4887d60838afb9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:57.312009 8202 raft_consensus.cc:2454] T 1b9e98043c5a470c891831081e3ce4f1 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 785d3ca35dcd4aea838abdfdf3584460 in term 1.
I20260928 02:34:57.312057 8334 raft_consensus.cc:2454] T 1b9e98043c5a470c891831081e3ce4f1 P a1187d9ed1b24dac9b4887d60838afb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 785d3ca35dcd4aea838abdfdf3584460 in term 1.
I20260928 02:34:57.312165 7994 leader_election.cc:302] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 785d3ca35dcd4aea838abdfdf3584460, ee5ade571f0349a1b4e3fa4edbeae2e5; no voters:
I20260928 02:34:57.312252 8426 raft_consensus.cc:2781] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:57.312316 8426 raft_consensus.cc:686] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460 [term 1 LEADER]: Becoming Leader. State: Replica: 785d3ca35dcd4aea838abdfdf3584460, State: Running, Role: LEADER
I20260928 02:34:57.312381 8426 consensus_queue.cc:237] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.312664 8434 tablet_replica.cc:439] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460: TxnStatusTablet state changed. Reason: New leader 785d3ca35dcd4aea838abdfdf3584460. Latest consensus state: current_term: 1 leader_uuid: "785d3ca35dcd4aea838abdfdf3584460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } } }
I20260928 02:34:57.312734 8434 tablet_replica.cc:442] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:34:57.312919 8456 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:34:57.313030 8456 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 02:34:57.312969 7902 catalog_manager.cc:5495] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460 reported cstate change: term changed from 0 to 1, leader changed from <none> to 785d3ca35dcd4aea838abdfdf3584460 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "785d3ca35dcd4aea838abdfdf3584460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } health_report { overall_health: UNKNOWN } } }
I20260928 02:34:57.333868 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:57.331451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47694 (local address 127.0.98.1:43567)
0928 02:34:57.331494 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:57.331496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:57.331523 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:34:57.331609 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:57.331610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:57.331620 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:57.331660 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:34:57.331777 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.332225 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.333289 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.333467 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.333510 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:57.333671 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:34:57.333695 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:34:57.333723 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:57.333797 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 02:34:57.333810 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:34:57.333920 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:57.331402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43567 (local address 127.0.0.1:47694)
0928 02:34:57.331471 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:57.331484 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:57.331542 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:57.331677 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:57.331678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:57.331680 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:34:57.331731 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:57.331733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.332281 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.332282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:57.333190 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:57.333193 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.333521 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.333522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:57.333585 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:57.333602 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:34:57.333736 (+ 134us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:34:57.333741 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:34:57.333795 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 02:34:57.333838 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 02:34:57.334794 8202 raft_consensus.cc:1261] T 1b9e98043c5a470c891831081e3ce4f1 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 1 FOLLOWER]: Refusing update from remote peer 785d3ca35dcd4aea838abdfdf3584460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:57.334793 8334 raft_consensus.cc:1261] T 1b9e98043c5a470c891831081e3ce4f1 P a1187d9ed1b24dac9b4887d60838afb9 [term 1 FOLLOWER]: Refusing update from remote peer 785d3ca35dcd4aea838abdfdf3584460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:57.334983 8434 consensus_queue.cc:1035] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:57.335085 8426 consensus_queue.cc:1035] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:57.335423 8421 tablet_replica.cc:439] T 1b9e98043c5a470c891831081e3ce4f1 P a1187d9ed1b24dac9b4887d60838afb9: TxnStatusTablet state changed. Reason: New leader 785d3ca35dcd4aea838abdfdf3584460. Latest consensus state: current_term: 1 leader_uuid: "785d3ca35dcd4aea838abdfdf3584460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } } }
I20260928 02:34:57.335520 8421 tablet_replica.cc:442] T 1b9e98043c5a470c891831081e3ce4f1 P a1187d9ed1b24dac9b4887d60838afb9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:34:57.335611 8420 tablet_replica.cc:439] T 1b9e98043c5a470c891831081e3ce4f1 P ee5ade571f0349a1b4e3fa4edbeae2e5: TxnStatusTablet state changed. Reason: New leader 785d3ca35dcd4aea838abdfdf3584460. Latest consensus state: current_term: 1 leader_uuid: "785d3ca35dcd4aea838abdfdf3584460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } } }
I20260928 02:34:57.335677 8420 tablet_replica.cc:442] T 1b9e98043c5a470c891831081e3ce4f1 P ee5ade571f0349a1b4e3fa4edbeae2e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:34:57.336318 8434 tablet_replica.cc:439] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "785d3ca35dcd4aea838abdfdf3584460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } } }
I20260928 02:34:57.336398 8434 tablet_replica.cc:442] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:34:57.336437 8434 tablet_replica.cc:439] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "785d3ca35dcd4aea838abdfdf3584460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } } }
I20260928 02:34:57.336479 8434 tablet_replica.cc:442] T 1b9e98043c5a470c891831081e3ce4f1 P 785d3ca35dcd4aea838abdfdf3584460: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:34:57.336596 8431 mvcc.cc:205] Tried to move back new op lower bound from 7334145627481956352 to 7334145627391930368. Current Snapshot: MvccSnapshot[applied={T|T < 7334145627481956352}]
I20260928 02:34:57.337154 8420 tablet_replica.cc:439] T 1b9e98043c5a470c891831081e3ce4f1 P ee5ade571f0349a1b4e3fa4edbeae2e5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "785d3ca35dcd4aea838abdfdf3584460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } } }
I20260928 02:34:57.337232 8420 tablet_replica.cc:442] T 1b9e98043c5a470c891831081e3ce4f1 P ee5ade571f0349a1b4e3fa4edbeae2e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:34:57.337502 8421 tablet_replica.cc:439] T 1b9e98043c5a470c891831081e3ce4f1 P a1187d9ed1b24dac9b4887d60838afb9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "785d3ca35dcd4aea838abdfdf3584460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } } }
I20260928 02:34:57.337579 8421 tablet_replica.cc:442] T 1b9e98043c5a470c891831081e3ce4f1 P a1187d9ed1b24dac9b4887d60838afb9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:34:57.341801 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:57.339389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42957 (local address 127.0.98.1:43567)
0928 02:34:57.339426 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:57.339428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:57.339523 (+ 95us) server_negotiation.cc:407] Connection header received
0928 02:34:57.339572 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:57.339573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:57.339581 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:57.339628 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:34:57.339701 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.340403 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.341061 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.341369 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.341432 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:57.341541 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:34:57.341570 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:34:57.341614 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:57.341645 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 02:34:57.341672 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":19}
I20260928 02:34:57.341789 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:57.339333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43567 (local address 127.0.98.1:42957)
0928 02:34:57.339463 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:57.339473 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:57.339522 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:57.339630 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:57.339631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:57.339632 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:34:57.339666 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:57.339667 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.340466 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.340466 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:57.341002 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:57.341003 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.341430 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.341430 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:57.341470 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:57.341481 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:34:57.341614 (+ 133us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:34:57.341614 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 02:34:57.341642 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 02:34:57.341691 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":14,"thread_start_us":68,"threads_started":1}
I20260928 02:34:57.344744 8334 raft_consensus.cc:1261] T 806b1f5f2d2c421893acbffcaafdc9a9 P a1187d9ed1b24dac9b4887d60838afb9 [term 1 FOLLOWER]: Refusing update from remote peer 785d3ca35dcd4aea838abdfdf3584460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:57.344928 8419 consensus_queue.cc:1035] T 806b1f5f2d2c421893acbffcaafdc9a9 P 785d3ca35dcd4aea838abdfdf3584460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:57.344980 8202 raft_consensus.cc:1261] T 806b1f5f2d2c421893acbffcaafdc9a9 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 1 FOLLOWER]: Refusing update from remote peer 785d3ca35dcd4aea838abdfdf3584460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:57.345160 8419 consensus_queue.cc:1035] T 806b1f5f2d2c421893acbffcaafdc9a9 P 785d3ca35dcd4aea838abdfdf3584460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:57.346479 8430 mvcc.cc:205] Tried to move back new op lower bound from 7334145627523051520 to 7334145626853851136. Current Snapshot: MvccSnapshot[applied={T|T < 7334145627523051520}]
I20260928 02:34:57.355093 7904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:34:57.355167 7904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:34:57.355715 7904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca69e07373f44b6096c5c7cd6b96bdd5: Sending DeleteTablet for 3 replicas of tablet 806b1f5f2d2c421893acbffcaafdc9a9
I20260928 02:34:57.355944 8041 tablet_service.cc:1514] Processing DeleteTablet for tablet 806b1f5f2d2c421893acbffcaafdc9a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47682
I20260928 02:34:57.355932 8322 tablet_service.cc:1514] Processing DeleteTablet for tablet 806b1f5f2d2c421893acbffcaafdc9a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32856
I20260928 02:34:57.355975 8182 tablet_service.cc:1514] Processing DeleteTablet for tablet 806b1f5f2d2c421893acbffcaafdc9a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40210
I20260928 02:34:57.356026 8441 tablet_replica.cc:330] T 806b1f5f2d2c421893acbffcaafdc9a9 P 785d3ca35dcd4aea838abdfdf3584460: stopping tablet replica
I20260928 02:34:57.356069 8441 raft_consensus.cc:2229] T 806b1f5f2d2c421893acbffcaafdc9a9 P 785d3ca35dcd4aea838abdfdf3584460 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:34:57.356120 8441 raft_consensus.cc:2258] T 806b1f5f2d2c421893acbffcaafdc9a9 P 785d3ca35dcd4aea838abdfdf3584460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:57.356171 8439 tablet_replica.cc:330] T 806b1f5f2d2c421893acbffcaafdc9a9 P a1187d9ed1b24dac9b4887d60838afb9: stopping tablet replica
I20260928 02:34:57.356206 8439 raft_consensus.cc:2229] T 806b1f5f2d2c421893acbffcaafdc9a9 P a1187d9ed1b24dac9b4887d60838afb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:34:57.356231 8439 raft_consensus.cc:2258] T 806b1f5f2d2c421893acbffcaafdc9a9 P a1187d9ed1b24dac9b4887d60838afb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:57.356285 8440 tablet_replica.cc:330] T 806b1f5f2d2c421893acbffcaafdc9a9 P ee5ade571f0349a1b4e3fa4edbeae2e5: stopping tablet replica
I20260928 02:34:57.356338 8440 raft_consensus.cc:2229] T 806b1f5f2d2c421893acbffcaafdc9a9 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:34:57.356391 8440 raft_consensus.cc:2258] T 806b1f5f2d2c421893acbffcaafdc9a9 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:57.356442 8441 ts_tablet_manager.cc:1905] T 806b1f5f2d2c421893acbffcaafdc9a9 P 785d3ca35dcd4aea838abdfdf3584460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:57.357970 8441 ts_tablet_manager.cc:1918] T 806b1f5f2d2c421893acbffcaafdc9a9 P 785d3ca35dcd4aea838abdfdf3584460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:34:57.358016 8441 log.cc:1196] T 806b1f5f2d2c421893acbffcaafdc9a9 P 785d3ca35dcd4aea838abdfdf3584460: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-0/wal/wals/806b1f5f2d2c421893acbffcaafdc9a9
I20260928 02:34:57.358199 8441 ts_tablet_manager.cc:1939] T 806b1f5f2d2c421893acbffcaafdc9a9 P 785d3ca35dcd4aea838abdfdf3584460: Deleting consensus metadata
I20260928 02:34:57.358494 8440 ts_tablet_manager.cc:1905] T 806b1f5f2d2c421893acbffcaafdc9a9 P ee5ade571f0349a1b4e3fa4edbeae2e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:57.358532 7889 catalog_manager.cc:4841] TS 785d3ca35dcd4aea838abdfdf3584460 (127.0.98.1:43567): tablet 806b1f5f2d2c421893acbffcaafdc9a9 (table test-table [id=bdce2262a3bb4a84a111d97d2a3dc31c]) successfully deleted
I20260928 02:34:57.359637 8440 ts_tablet_manager.cc:1918] T 806b1f5f2d2c421893acbffcaafdc9a9 P ee5ade571f0349a1b4e3fa4edbeae2e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:34:57.359686 8440 log.cc:1196] T 806b1f5f2d2c421893acbffcaafdc9a9 P ee5ade571f0349a1b4e3fa4edbeae2e5: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-1/wal/wals/806b1f5f2d2c421893acbffcaafdc9a9
I20260928 02:34:57.359855 8440 ts_tablet_manager.cc:1939] T 806b1f5f2d2c421893acbffcaafdc9a9 P ee5ade571f0349a1b4e3fa4edbeae2e5: Deleting consensus metadata
I20260928 02:34:57.361155 7890 catalog_manager.cc:4841] TS ee5ade571f0349a1b4e3fa4edbeae2e5 (127.0.98.2:42811): tablet 806b1f5f2d2c421893acbffcaafdc9a9 (table test-table [id=bdce2262a3bb4a84a111d97d2a3dc31c]) successfully deleted
I20260928 02:34:57.361439 8439 ts_tablet_manager.cc:1905] T 806b1f5f2d2c421893acbffcaafdc9a9 P a1187d9ed1b24dac9b4887d60838afb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:57.362813 8439 ts_tablet_manager.cc:1918] T 806b1f5f2d2c421893acbffcaafdc9a9 P a1187d9ed1b24dac9b4887d60838afb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:34:57.362862 8439 log.cc:1196] T 806b1f5f2d2c421893acbffcaafdc9a9 P a1187d9ed1b24dac9b4887d60838afb9: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-2/wal/wals/806b1f5f2d2c421893acbffcaafdc9a9
I20260928 02:34:57.363044 8439 ts_tablet_manager.cc:1939] T 806b1f5f2d2c421893acbffcaafdc9a9 P a1187d9ed1b24dac9b4887d60838afb9: Deleting consensus metadata
I20260928 02:34:57.363291 7889 catalog_manager.cc:4841] TS a1187d9ed1b24dac9b4887d60838afb9 (127.0.98.3:35633): tablet 806b1f5f2d2c421893acbffcaafdc9a9 (table test-table [id=bdce2262a3bb4a84a111d97d2a3dc31c]) successfully deleted
Sep 28 02:34:57 dist-test-slave-fz47 krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:34:57 dist-test-slave-fz47 krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:34:57 dist-test-slave-fz47 krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:34:57.384660 7954 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:34:57.377302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38304 (local address 127.0.98.62:35019)
0928 02:34:57.377342 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:57.377345 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:57.378778 (+ 1433us) server_negotiation.cc:407] Connection header received
0928 02:34:57.378849 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:57.378849 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:57.378860 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:57.378901 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:57.379610 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.380462 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.381612 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.381821 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.381877 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:57.384599 (+ 2722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38304: BlockingRecv error: recv got EOF from 127.0.0.1:38304 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 28 02:34:57 dist-test-slave-fz47 krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 02:34:57.391486 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:57.385493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38314 (local address 127.0.98.62:35019)
0928 02:34:57.385530 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:57.385531 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:57.385633 (+ 102us) server_negotiation.cc:407] Connection header received
0928 02:34:57.385687 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:57.385688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:57.385697 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:57.385741 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:57.386153 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.386652 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.387286 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.387459 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.387509 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:57.389375 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:57.389376 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:57.389377 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:57.389384 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:57.390570 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:57.390871 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:57.390872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:57.390872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:57.390917 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:57.391069 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:57.391069 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:57.391070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:57.391165 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:57.391246 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:57.391335 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 02:34:57.391402 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:34:57.392921 7904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:34:57.393076 7904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:34:57.394381 8041 tablet_service.cc:1467] Processing CreateTablet for tablet d23844a9e2da430f87327ebaa5e4c2f0 (DEFAULT_TABLE table=test-table [id=ba93afe2239c43a79b28fd10014a92d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:57.394398 8322 tablet_service.cc:1467] Processing CreateTablet for tablet d23844a9e2da430f87327ebaa5e4c2f0 (DEFAULT_TABLE table=test-table [id=ba93afe2239c43a79b28fd10014a92d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:57.394520 8322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d23844a9e2da430f87327ebaa5e4c2f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:57.394515 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d23844a9e2da430f87327ebaa5e4c2f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:57.395830 8414 tablet_bootstrap.cc:492] T d23844a9e2da430f87327ebaa5e4c2f0 P a1187d9ed1b24dac9b4887d60838afb9: Bootstrap starting.
I20260928 02:34:57.395885 8182 tablet_service.cc:1467] Processing CreateTablet for tablet d23844a9e2da430f87327ebaa5e4c2f0 (DEFAULT_TABLE table=test-table [id=ba93afe2239c43a79b28fd10014a92d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:57.396005 8182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d23844a9e2da430f87327ebaa5e4c2f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:57.396400 8414 tablet_bootstrap.cc:654] T d23844a9e2da430f87327ebaa5e4c2f0 P a1187d9ed1b24dac9b4887d60838afb9: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:57.397292 8417 tablet_bootstrap.cc:492] T d23844a9e2da430f87327ebaa5e4c2f0 P ee5ade571f0349a1b4e3fa4edbeae2e5: Bootstrap starting.
I20260928 02:34:57.397322 8413 tablet_bootstrap.cc:492] T d23844a9e2da430f87327ebaa5e4c2f0 P 785d3ca35dcd4aea838abdfdf3584460: Bootstrap starting.
I20260928 02:34:57.397596 8413 tablet_bootstrap.cc:654] T d23844a9e2da430f87327ebaa5e4c2f0 P 785d3ca35dcd4aea838abdfdf3584460: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:57.397637 8417 tablet_bootstrap.cc:654] T d23844a9e2da430f87327ebaa5e4c2f0 P ee5ade571f0349a1b4e3fa4edbeae2e5: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:57.398136 8413 tablet_bootstrap.cc:492] T d23844a9e2da430f87327ebaa5e4c2f0 P 785d3ca35dcd4aea838abdfdf3584460: No bootstrap required, opened a new log
I20260928 02:34:57.398193 8413 ts_tablet_manager.cc:1397] T d23844a9e2da430f87327ebaa5e4c2f0 P 785d3ca35dcd4aea838abdfdf3584460: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:34:57.398272 8417 tablet_bootstrap.cc:492] T d23844a9e2da430f87327ebaa5e4c2f0 P ee5ade571f0349a1b4e3fa4edbeae2e5: No bootstrap required, opened a new log
I20260928 02:34:57.398319 8417 ts_tablet_manager.cc:1397] T d23844a9e2da430f87327ebaa5e4c2f0 P ee5ade571f0349a1b4e3fa4edbeae2e5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:34:57.398299 8413 raft_consensus.cc:348] T d23844a9e2da430f87327ebaa5e4c2f0 P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.398373 8413 raft_consensus.cc:374] T d23844a9e2da430f87327ebaa5e4c2f0 P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:57.398393 8413 raft_consensus.cc:729] T d23844a9e2da430f87327ebaa5e4c2f0 P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 785d3ca35dcd4aea838abdfdf3584460, State: Initialized, Role: FOLLOWER
I20260928 02:34:57.398433 8413 consensus_queue.cc:260] T d23844a9e2da430f87327ebaa5e4c2f0 P 785d3ca35dcd4aea838abdfdf3584460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.398455 8417 raft_consensus.cc:348] T d23844a9e2da430f87327ebaa5e4c2f0 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.398520 8413 ts_tablet_manager.cc:1428] T d23844a9e2da430f87327ebaa5e4c2f0 P 785d3ca35dcd4aea838abdfdf3584460: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:57.398540 8417 raft_consensus.cc:374] T d23844a9e2da430f87327ebaa5e4c2f0 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:57.398576 8417 raft_consensus.cc:729] T d23844a9e2da430f87327ebaa5e4c2f0 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee5ade571f0349a1b4e3fa4edbeae2e5, State: Initialized, Role: FOLLOWER
I20260928 02:34:57.398607 8417 consensus_queue.cc:260] T d23844a9e2da430f87327ebaa5e4c2f0 P ee5ade571f0349a1b4e3fa4edbeae2e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.398689 8417 ts_tablet_manager.cc:1428] T d23844a9e2da430f87327ebaa5e4c2f0 P ee5ade571f0349a1b4e3fa4edbeae2e5: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:57.398847 8414 tablet_bootstrap.cc:492] T d23844a9e2da430f87327ebaa5e4c2f0 P a1187d9ed1b24dac9b4887d60838afb9: No bootstrap required, opened a new log
I20260928 02:34:57.398901 8414 ts_tablet_manager.cc:1397] T d23844a9e2da430f87327ebaa5e4c2f0 P a1187d9ed1b24dac9b4887d60838afb9: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 02:34:57.399063 8414 raft_consensus.cc:348] T d23844a9e2da430f87327ebaa5e4c2f0 P a1187d9ed1b24dac9b4887d60838afb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.399132 8414 raft_consensus.cc:374] T d23844a9e2da430f87327ebaa5e4c2f0 P a1187d9ed1b24dac9b4887d60838afb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:57.399151 8414 raft_consensus.cc:729] T d23844a9e2da430f87327ebaa5e4c2f0 P a1187d9ed1b24dac9b4887d60838afb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1187d9ed1b24dac9b4887d60838afb9, State: Initialized, Role: FOLLOWER
I20260928 02:34:57.399191 8414 consensus_queue.cc:260] T d23844a9e2da430f87327ebaa5e4c2f0 P a1187d9ed1b24dac9b4887d60838afb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.399288 8414 ts_tablet_manager.cc:1428] T d23844a9e2da430f87327ebaa5e4c2f0 P a1187d9ed1b24dac9b4887d60838afb9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:57.478904 8419 raft_consensus.cc:482] T d23844a9e2da430f87327ebaa5e4c2f0 P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:34:57.478996 8419 raft_consensus.cc:504] T d23844a9e2da430f87327ebaa5e4c2f0 P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.479139 8419 leader_election.cc:288] T d23844a9e2da430f87327ebaa5e4c2f0 P 785d3ca35dcd4aea838abdfdf3584460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee5ade571f0349a1b4e3fa4edbeae2e5 (127.0.98.2:42811), a1187d9ed1b24dac9b4887d60838afb9 (127.0.98.3:35633)
I20260928 02:34:57.479318 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d23844a9e2da430f87327ebaa5e4c2f0" candidate_uuid: "785d3ca35dcd4aea838abdfdf3584460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" is_pre_election: true
I20260928 02:34:57.479316 8334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d23844a9e2da430f87327ebaa5e4c2f0" candidate_uuid: "785d3ca35dcd4aea838abdfdf3584460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1187d9ed1b24dac9b4887d60838afb9" is_pre_election: true
I20260928 02:34:57.479401 8334 raft_consensus.cc:2454] T d23844a9e2da430f87327ebaa5e4c2f0 P a1187d9ed1b24dac9b4887d60838afb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 785d3ca35dcd4aea838abdfdf3584460 in term 0.
I20260928 02:34:57.479400 8202 raft_consensus.cc:2454] T d23844a9e2da430f87327ebaa5e4c2f0 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 785d3ca35dcd4aea838abdfdf3584460 in term 0.
I20260928 02:34:57.479545 7993 leader_election.cc:302] T d23844a9e2da430f87327ebaa5e4c2f0 P 785d3ca35dcd4aea838abdfdf3584460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 785d3ca35dcd4aea838abdfdf3584460, a1187d9ed1b24dac9b4887d60838afb9; no voters:
I20260928 02:34:57.479643 8419 raft_consensus.cc:2781] T d23844a9e2da430f87327ebaa5e4c2f0 P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:34:57.479681 8419 raft_consensus.cc:482] T d23844a9e2da430f87327ebaa5e4c2f0 P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:34:57.479698 8419 raft_consensus.cc:3037] T d23844a9e2da430f87327ebaa5e4c2f0 P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:57.480317 8419 raft_consensus.cc:504] T d23844a9e2da430f87327ebaa5e4c2f0 P 785d3ca35dcd4aea838abdfdf3584460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.480422 8419 leader_election.cc:288] T d23844a9e2da430f87327ebaa5e4c2f0 P 785d3ca35dcd4aea838abdfdf3584460 [CANDIDATE]: Term 1 election: Requested vote from peers ee5ade571f0349a1b4e3fa4edbeae2e5 (127.0.98.2:42811), a1187d9ed1b24dac9b4887d60838afb9 (127.0.98.3:35633)
I20260928 02:34:57.480583 8334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d23844a9e2da430f87327ebaa5e4c2f0" candidate_uuid: "785d3ca35dcd4aea838abdfdf3584460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1187d9ed1b24dac9b4887d60838afb9"
I20260928 02:34:57.480612 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d23844a9e2da430f87327ebaa5e4c2f0" candidate_uuid: "785d3ca35dcd4aea838abdfdf3584460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5"
I20260928 02:34:57.480650 8334 raft_consensus.cc:3037] T d23844a9e2da430f87327ebaa5e4c2f0 P a1187d9ed1b24dac9b4887d60838afb9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:57.480674 8202 raft_consensus.cc:3037] T d23844a9e2da430f87327ebaa5e4c2f0 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:57.481367 8334 raft_consensus.cc:2454] T d23844a9e2da430f87327ebaa5e4c2f0 P a1187d9ed1b24dac9b4887d60838afb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 785d3ca35dcd4aea838abdfdf3584460 in term 1.
I20260928 02:34:57.481367 8202 raft_consensus.cc:2454] T d23844a9e2da430f87327ebaa5e4c2f0 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 785d3ca35dcd4aea838abdfdf3584460 in term 1.
I20260928 02:34:57.481524 7993 leader_election.cc:302] T d23844a9e2da430f87327ebaa5e4c2f0 P 785d3ca35dcd4aea838abdfdf3584460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 785d3ca35dcd4aea838abdfdf3584460, a1187d9ed1b24dac9b4887d60838afb9; no voters:
I20260928 02:34:57.481621 8419 raft_consensus.cc:2781] T d23844a9e2da430f87327ebaa5e4c2f0 P 785d3ca35dcd4aea838abdfdf3584460 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:57.481678 8419 raft_consensus.cc:686] T d23844a9e2da430f87327ebaa5e4c2f0 P 785d3ca35dcd4aea838abdfdf3584460 [term 1 LEADER]: Becoming Leader. State: Replica: 785d3ca35dcd4aea838abdfdf3584460, State: Running, Role: LEADER
I20260928 02:34:57.481717 8419 consensus_queue.cc:237] T d23844a9e2da430f87327ebaa5e4c2f0 P 785d3ca35dcd4aea838abdfdf3584460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.482189 7905 catalog_manager.cc:5495] T d23844a9e2da430f87327ebaa5e4c2f0 P 785d3ca35dcd4aea838abdfdf3584460 reported cstate change: term changed from 0 to 1, leader changed from <none> to 785d3ca35dcd4aea838abdfdf3584460 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "785d3ca35dcd4aea838abdfdf3584460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } health_report { overall_health: UNKNOWN } } }
I20260928 02:34:57.490239 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:57.488421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47698 (local address 127.0.98.1:43567)
0928 02:34:57.488463 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:57.488464 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:57.488472 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:34:57.488506 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:57.488507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:57.488514 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:57.488560 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:34:57.488641 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.489097 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.489799 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.489972 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.490015 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:57.490101 (+ 86us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:34:57.490122 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:34:57.490151 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:57.490182 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 02:34:57.490194 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 02:34:57.490830 8202 raft_consensus.cc:1261] T d23844a9e2da430f87327ebaa5e4c2f0 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 1 FOLLOWER]: Refusing update from remote peer 785d3ca35dcd4aea838abdfdf3584460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:57.490851 8334 raft_consensus.cc:1261] T d23844a9e2da430f87327ebaa5e4c2f0 P a1187d9ed1b24dac9b4887d60838afb9 [term 1 FOLLOWER]: Refusing update from remote peer 785d3ca35dcd4aea838abdfdf3584460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:57.491026 8426 consensus_queue.cc:1035] T d23844a9e2da430f87327ebaa5e4c2f0 P 785d3ca35dcd4aea838abdfdf3584460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:57.491109 8419 consensus_queue.cc:1035] T d23844a9e2da430f87327ebaa5e4c2f0 P 785d3ca35dcd4aea838abdfdf3584460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:57.493235 7902 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:34:57.493337 7902 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:34:57.493804 7902 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca69e07373f44b6096c5c7cd6b96bdd5: Sending DeleteTablet for 3 replicas of tablet d23844a9e2da430f87327ebaa5e4c2f0
I20260928 02:34:57.494026 8041 tablet_service.cc:1514] Processing DeleteTablet for tablet d23844a9e2da430f87327ebaa5e4c2f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47682
I20260928 02:34:57.494115 8182 tablet_service.cc:1514] Processing DeleteTablet for tablet d23844a9e2da430f87327ebaa5e4c2f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40210
I20260928 02:34:57.494109 8322 tablet_service.cc:1514] Processing DeleteTablet for tablet d23844a9e2da430f87327ebaa5e4c2f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32856
I20260928 02:34:57.494124 8441 tablet_replica.cc:330] T d23844a9e2da430f87327ebaa5e4c2f0 P 785d3ca35dcd4aea838abdfdf3584460: stopping tablet replica
I20260928 02:34:57.494174 8441 raft_consensus.cc:2229] T d23844a9e2da430f87327ebaa5e4c2f0 P 785d3ca35dcd4aea838abdfdf3584460 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:34:57.494184 8440 tablet_replica.cc:330] T d23844a9e2da430f87327ebaa5e4c2f0 P ee5ade571f0349a1b4e3fa4edbeae2e5: stopping tablet replica
I20260928 02:34:57.494182 8439 tablet_replica.cc:330] T d23844a9e2da430f87327ebaa5e4c2f0 P a1187d9ed1b24dac9b4887d60838afb9: stopping tablet replica
I20260928 02:34:57.494220 8441 raft_consensus.cc:2258] T d23844a9e2da430f87327ebaa5e4c2f0 P 785d3ca35dcd4aea838abdfdf3584460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:57.494220 8439 raft_consensus.cc:2229] T d23844a9e2da430f87327ebaa5e4c2f0 P a1187d9ed1b24dac9b4887d60838afb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:34:57.494220 8440 raft_consensus.cc:2229] T d23844a9e2da430f87327ebaa5e4c2f0 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:34:57.494254 8440 raft_consensus.cc:2258] T d23844a9e2da430f87327ebaa5e4c2f0 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:57.494292 8439 raft_consensus.cc:2258] T d23844a9e2da430f87327ebaa5e4c2f0 P a1187d9ed1b24dac9b4887d60838afb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:57.494328 7902 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:34:57.494443 7902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:34:57.494573 8439 ts_tablet_manager.cc:1905] T d23844a9e2da430f87327ebaa5e4c2f0 P a1187d9ed1b24dac9b4887d60838afb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:57.494648 8441 ts_tablet_manager.cc:1905] T d23844a9e2da430f87327ebaa5e4c2f0 P 785d3ca35dcd4aea838abdfdf3584460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:57.494810 8440 ts_tablet_manager.cc:1905] T d23844a9e2da430f87327ebaa5e4c2f0 P ee5ade571f0349a1b4e3fa4edbeae2e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:57.495803 8439 ts_tablet_manager.cc:1918] T d23844a9e2da430f87327ebaa5e4c2f0 P a1187d9ed1b24dac9b4887d60838afb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:34:57.495839 8439 log.cc:1196] T d23844a9e2da430f87327ebaa5e4c2f0 P a1187d9ed1b24dac9b4887d60838afb9: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-2/wal/wals/d23844a9e2da430f87327ebaa5e4c2f0
I20260928 02:34:57.495998 8439 ts_tablet_manager.cc:1939] T d23844a9e2da430f87327ebaa5e4c2f0 P a1187d9ed1b24dac9b4887d60838afb9: Deleting consensus metadata
I20260928 02:34:57.496007 8441 ts_tablet_manager.cc:1918] T d23844a9e2da430f87327ebaa5e4c2f0 P 785d3ca35dcd4aea838abdfdf3584460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:34:57.496007 8440 ts_tablet_manager.cc:1918] T d23844a9e2da430f87327ebaa5e4c2f0 P ee5ade571f0349a1b4e3fa4edbeae2e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:34:57.496053 8441 log.cc:1196] T d23844a9e2da430f87327ebaa5e4c2f0 P 785d3ca35dcd4aea838abdfdf3584460: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-0/wal/wals/d23844a9e2da430f87327ebaa5e4c2f0
I20260928 02:34:57.496053 8440 log.cc:1196] T d23844a9e2da430f87327ebaa5e4c2f0 P ee5ade571f0349a1b4e3fa4edbeae2e5: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-1/wal/wals/d23844a9e2da430f87327ebaa5e4c2f0
I20260928 02:34:57.496212 7889 catalog_manager.cc:4841] TS a1187d9ed1b24dac9b4887d60838afb9 (127.0.98.3:35633): tablet d23844a9e2da430f87327ebaa5e4c2f0 (table test-table [id=ba93afe2239c43a79b28fd10014a92d2]) successfully deleted
I20260928 02:34:57.496236 8440 ts_tablet_manager.cc:1939] T d23844a9e2da430f87327ebaa5e4c2f0 P ee5ade571f0349a1b4e3fa4edbeae2e5: Deleting consensus metadata
I20260928 02:34:57.496243 8441 ts_tablet_manager.cc:1939] T d23844a9e2da430f87327ebaa5e4c2f0 P 785d3ca35dcd4aea838abdfdf3584460: Deleting consensus metadata
I20260928 02:34:57.496583 7890 catalog_manager.cc:4841] TS ee5ade571f0349a1b4e3fa4edbeae2e5 (127.0.98.2:42811): tablet d23844a9e2da430f87327ebaa5e4c2f0 (table test-table [id=ba93afe2239c43a79b28fd10014a92d2]) successfully deleted
I20260928 02:34:57.496683 7889 catalog_manager.cc:4841] TS 785d3ca35dcd4aea838abdfdf3584460 (127.0.98.1:43567): tablet d23844a9e2da430f87327ebaa5e4c2f0 (table test-table [id=ba93afe2239c43a79b28fd10014a92d2]) successfully deleted
I20260928 02:34:57.497195 8322 tablet_service.cc:1467] Processing CreateTablet for tablet 62f5d331889e4feda04688c333703112 (DEFAULT_TABLE table=test-table [id=bbc1fa6c2ca649e392b1d6953a07a1cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:57.497189 8041 tablet_service.cc:1467] Processing CreateTablet for tablet 62f5d331889e4feda04688c333703112 (DEFAULT_TABLE table=test-table [id=bbc1fa6c2ca649e392b1d6953a07a1cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:57.497211 8182 tablet_service.cc:1467] Processing CreateTablet for tablet 62f5d331889e4feda04688c333703112 (DEFAULT_TABLE table=test-table [id=bbc1fa6c2ca649e392b1d6953a07a1cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:34:57.497311 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62f5d331889e4feda04688c333703112. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:57.497310 8322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62f5d331889e4feda04688c333703112. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:57.497324 8182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62f5d331889e4feda04688c333703112. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:57.498229 8417 tablet_bootstrap.cc:492] T 62f5d331889e4feda04688c333703112 P ee5ade571f0349a1b4e3fa4edbeae2e5: Bootstrap starting.
I20260928 02:34:57.498358 8413 tablet_bootstrap.cc:492] T 62f5d331889e4feda04688c333703112 P 785d3ca35dcd4aea838abdfdf3584460: Bootstrap starting.
I20260928 02:34:57.498363 8414 tablet_bootstrap.cc:492] T 62f5d331889e4feda04688c333703112 P a1187d9ed1b24dac9b4887d60838afb9: Bootstrap starting.
I20260928 02:34:57.498636 8417 tablet_bootstrap.cc:654] T 62f5d331889e4feda04688c333703112 P ee5ade571f0349a1b4e3fa4edbeae2e5: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:57.498636 8413 tablet_bootstrap.cc:654] T 62f5d331889e4feda04688c333703112 P 785d3ca35dcd4aea838abdfdf3584460: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:57.498694 8414 tablet_bootstrap.cc:654] T 62f5d331889e4feda04688c333703112 P a1187d9ed1b24dac9b4887d60838afb9: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:57.499208 8417 tablet_bootstrap.cc:492] T 62f5d331889e4feda04688c333703112 P ee5ade571f0349a1b4e3fa4edbeae2e5: No bootstrap required, opened a new log
I20260928 02:34:57.499253 8417 ts_tablet_manager.cc:1397] T 62f5d331889e4feda04688c333703112 P ee5ade571f0349a1b4e3fa4edbeae2e5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:34:57.499372 8414 tablet_bootstrap.cc:492] T 62f5d331889e4feda04688c333703112 P a1187d9ed1b24dac9b4887d60838afb9: No bootstrap required, opened a new log
I20260928 02:34:57.499377 8417 raft_consensus.cc:348] T 62f5d331889e4feda04688c333703112 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.499408 8414 ts_tablet_manager.cc:1397] T 62f5d331889e4feda04688c333703112 P a1187d9ed1b24dac9b4887d60838afb9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 02:34:57.499426 8417 raft_consensus.cc:374] T 62f5d331889e4feda04688c333703112 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:57.499452 8417 raft_consensus.cc:729] T 62f5d331889e4feda04688c333703112 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee5ade571f0349a1b4e3fa4edbeae2e5, State: Initialized, Role: FOLLOWER
I20260928 02:34:57.499496 8417 consensus_queue.cc:260] T 62f5d331889e4feda04688c333703112 P ee5ade571f0349a1b4e3fa4edbeae2e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.499545 8414 raft_consensus.cc:348] T 62f5d331889e4feda04688c333703112 P a1187d9ed1b24dac9b4887d60838afb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.499600 8417 ts_tablet_manager.cc:1428] T 62f5d331889e4feda04688c333703112 P ee5ade571f0349a1b4e3fa4edbeae2e5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:57.499607 8414 raft_consensus.cc:374] T 62f5d331889e4feda04688c333703112 P a1187d9ed1b24dac9b4887d60838afb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:57.499630 8414 raft_consensus.cc:729] T 62f5d331889e4feda04688c333703112 P a1187d9ed1b24dac9b4887d60838afb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1187d9ed1b24dac9b4887d60838afb9, State: Initialized, Role: FOLLOWER
I20260928 02:34:57.499675 8414 consensus_queue.cc:260] T 62f5d331889e4feda04688c333703112 P a1187d9ed1b24dac9b4887d60838afb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.499752 8414 ts_tablet_manager.cc:1428] T 62f5d331889e4feda04688c333703112 P a1187d9ed1b24dac9b4887d60838afb9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:57.499984 8413 tablet_bootstrap.cc:492] T 62f5d331889e4feda04688c333703112 P 785d3ca35dcd4aea838abdfdf3584460: No bootstrap required, opened a new log
I20260928 02:34:57.500033 8413 ts_tablet_manager.cc:1397] T 62f5d331889e4feda04688c333703112 P 785d3ca35dcd4aea838abdfdf3584460: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:34:57.500141 8413 raft_consensus.cc:348] T 62f5d331889e4feda04688c333703112 P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.500208 8413 raft_consensus.cc:374] T 62f5d331889e4feda04688c333703112 P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:57.500226 8413 raft_consensus.cc:729] T 62f5d331889e4feda04688c333703112 P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 785d3ca35dcd4aea838abdfdf3584460, State: Initialized, Role: FOLLOWER
I20260928 02:34:57.500264 8413 consensus_queue.cc:260] T 62f5d331889e4feda04688c333703112 P 785d3ca35dcd4aea838abdfdf3584460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.500347 8413 ts_tablet_manager.cc:1428] T 62f5d331889e4feda04688c333703112 P 785d3ca35dcd4aea838abdfdf3584460: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:57.564122 8421 raft_consensus.cc:482] T 62f5d331889e4feda04688c333703112 P a1187d9ed1b24dac9b4887d60838afb9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:34:57.564213 8421 raft_consensus.cc:504] T 62f5d331889e4feda04688c333703112 P a1187d9ed1b24dac9b4887d60838afb9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.564383 8421 leader_election.cc:288] T 62f5d331889e4feda04688c333703112 P a1187d9ed1b24dac9b4887d60838afb9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 785d3ca35dcd4aea838abdfdf3584460 (127.0.98.1:43567), ee5ade571f0349a1b4e3fa4edbeae2e5 (127.0.98.2:42811)
I20260928 02:34:57.564570 8062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62f5d331889e4feda04688c333703112" candidate_uuid: "a1187d9ed1b24dac9b4887d60838afb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "785d3ca35dcd4aea838abdfdf3584460" is_pre_election: true
I20260928 02:34:57.564570 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62f5d331889e4feda04688c333703112" candidate_uuid: "a1187d9ed1b24dac9b4887d60838afb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" is_pre_election: true
I20260928 02:34:57.564677 8202 raft_consensus.cc:2454] T 62f5d331889e4feda04688c333703112 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1187d9ed1b24dac9b4887d60838afb9 in term 0.
I20260928 02:34:57.564690 8062 raft_consensus.cc:2454] T 62f5d331889e4feda04688c333703112 P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1187d9ed1b24dac9b4887d60838afb9 in term 0.
I20260928 02:34:57.564836 8274 leader_election.cc:302] T 62f5d331889e4feda04688c333703112 P a1187d9ed1b24dac9b4887d60838afb9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1187d9ed1b24dac9b4887d60838afb9, ee5ade571f0349a1b4e3fa4edbeae2e5; no voters:
I20260928 02:34:57.564944 8421 raft_consensus.cc:2781] T 62f5d331889e4feda04688c333703112 P a1187d9ed1b24dac9b4887d60838afb9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:34:57.564991 8421 raft_consensus.cc:482] T 62f5d331889e4feda04688c333703112 P a1187d9ed1b24dac9b4887d60838afb9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:34:57.565004 8421 raft_consensus.cc:3037] T 62f5d331889e4feda04688c333703112 P a1187d9ed1b24dac9b4887d60838afb9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:57.565670 8421 raft_consensus.cc:504] T 62f5d331889e4feda04688c333703112 P a1187d9ed1b24dac9b4887d60838afb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.565778 8421 leader_election.cc:288] T 62f5d331889e4feda04688c333703112 P a1187d9ed1b24dac9b4887d60838afb9 [CANDIDATE]: Term 1 election: Requested vote from peers 785d3ca35dcd4aea838abdfdf3584460 (127.0.98.1:43567), ee5ade571f0349a1b4e3fa4edbeae2e5 (127.0.98.2:42811)
I20260928 02:34:57.565922 8062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62f5d331889e4feda04688c333703112" candidate_uuid: "a1187d9ed1b24dac9b4887d60838afb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "785d3ca35dcd4aea838abdfdf3584460"
I20260928 02:34:57.565943 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62f5d331889e4feda04688c333703112" candidate_uuid: "a1187d9ed1b24dac9b4887d60838afb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5"
I20260928 02:34:57.566004 8202 raft_consensus.cc:3037] T 62f5d331889e4feda04688c333703112 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:57.566001 8062 raft_consensus.cc:3037] T 62f5d331889e4feda04688c333703112 P 785d3ca35dcd4aea838abdfdf3584460 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:57.566524 8202 raft_consensus.cc:2454] T 62f5d331889e4feda04688c333703112 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1187d9ed1b24dac9b4887d60838afb9 in term 1.
I20260928 02:34:57.566659 8274 leader_election.cc:302] T 62f5d331889e4feda04688c333703112 P a1187d9ed1b24dac9b4887d60838afb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1187d9ed1b24dac9b4887d60838afb9, ee5ade571f0349a1b4e3fa4edbeae2e5; no voters:
I20260928 02:34:57.566670 8062 raft_consensus.cc:2454] T 62f5d331889e4feda04688c333703112 P 785d3ca35dcd4aea838abdfdf3584460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1187d9ed1b24dac9b4887d60838afb9 in term 1.
I20260928 02:34:57.566743 8421 raft_consensus.cc:2781] T 62f5d331889e4feda04688c333703112 P a1187d9ed1b24dac9b4887d60838afb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:57.566856 8421 raft_consensus.cc:686] T 62f5d331889e4feda04688c333703112 P a1187d9ed1b24dac9b4887d60838afb9 [term 1 LEADER]: Becoming Leader. State: Replica: a1187d9ed1b24dac9b4887d60838afb9, State: Running, Role: LEADER
I20260928 02:34:57.566924 8421 consensus_queue.cc:237] T 62f5d331889e4feda04688c333703112 P a1187d9ed1b24dac9b4887d60838afb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } }
I20260928 02:34:57.567549 7905 catalog_manager.cc:5495] T 62f5d331889e4feda04688c333703112 P a1187d9ed1b24dac9b4887d60838afb9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1187d9ed1b24dac9b4887d60838afb9 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a1187d9ed1b24dac9b4887d60838afb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1187d9ed1b24dac9b4887d60838afb9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35633 } health_report { overall_health: HEALTHY } } }
I20260928 02:34:57.574112 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:57.572236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32872 (local address 127.0.98.3:35633)
0928 02:34:57.572283 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:57.572284 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:57.572291 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:34:57.572316 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:57.572317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:57.572333 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:57.572388 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:34:57.572476 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.572926 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.573555 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.573728 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.573771 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:57.573858 (+ 87us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:34:57.573954 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:34:57.573999 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:57.574014 (+ 15us) server_negotiation.cc:299] Negotiation successful
0928 02:34:57.574030 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 02:34:57.577932 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:57.576032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34293 (local address 127.0.98.1:43567)
0928 02:34:57.576069 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:57.576071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:57.576159 (+ 88us) server_negotiation.cc:407] Connection header received
0928 02:34:57.576200 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:57.576201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:57.576208 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:57.576253 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:34:57.576329 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.576745 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.577455 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.577626 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.577669 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:57.577783 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:34:57.577804 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:34:57.577833 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:57.577873 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 02:34:57.577884 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:34:57.577963 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:57.575972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43567 (local address 127.0.98.3:34293)
0928 02:34:57.576109 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:57.576119 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:57.576166 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:57.576256 (+ 90us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:57.576257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:57.576257 (+ 0us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:34:57.576298 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:57.576299 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.576793 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.576793 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:57.577401 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:57.577403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.577675 (+ 272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.577676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:57.577717 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:57.577727 (+ 10us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:34:57.577842 (+ 115us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:34:57.577843 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 02:34:57.577871 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 02:34:57.577900 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":52,"threads_started":1}
I20260928 02:34:57.579597 8202 raft_consensus.cc:1261] T 62f5d331889e4feda04688c333703112 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 1 FOLLOWER]: Refusing update from remote peer a1187d9ed1b24dac9b4887d60838afb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:57.579668 8062 raft_consensus.cc:1261] T 62f5d331889e4feda04688c333703112 P 785d3ca35dcd4aea838abdfdf3584460 [term 1 FOLLOWER]: Refusing update from remote peer a1187d9ed1b24dac9b4887d60838afb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:34:57.579844 8489 consensus_queue.cc:1035] T 62f5d331889e4feda04688c333703112 P a1187d9ed1b24dac9b4887d60838afb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee5ade571f0349a1b4e3fa4edbeae2e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:57.579952 8421 consensus_queue.cc:1035] T 62f5d331889e4feda04688c333703112 P a1187d9ed1b24dac9b4887d60838afb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "785d3ca35dcd4aea838abdfdf3584460" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:34:57.586066 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:57.583972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51569 (local address 127.0.98.3:35633)
0928 02:34:57.584071 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:57.584072 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:57.584078 (+ 6us) server_negotiation.cc:407] Connection header received
0928 02:34:57.584096 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:57.584097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:57.584106 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:57.584146 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:34:57.584276 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.584748 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.585462 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.585662 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.585717 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:57.585867 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:34:57.585896 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:34:57.585936 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:57.585994 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 02:34:57.586008 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 02:34:57.586097 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:57.583925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35633 (local address 127.0.98.1:51569)
0928 02:34:57.583970 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:57.583980 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:34:57.584015 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:34:57.584177 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:34:57.584178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:34:57.584179 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:34:57.584211 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:57.584212 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.584809 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.584810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:57.585410 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:34:57.585412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:34:57.585715 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:34:57.585716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:34:57.585779 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:57.585785 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:34:57.585949 (+ 164us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:34:57.585952 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:34:57.585993 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 02:34:57.586038 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 02:34:57.592463 7902 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:34:57.592573 7902 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:34:57.593119 7902 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca69e07373f44b6096c5c7cd6b96bdd5: Sending DeleteTablet for 3 replicas of tablet 62f5d331889e4feda04688c333703112
I20260928 02:34:57.593371 8041 tablet_service.cc:1514] Processing DeleteTablet for tablet 62f5d331889e4feda04688c333703112 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47682
I20260928 02:34:57.593458 8441 tablet_replica.cc:330] T 62f5d331889e4feda04688c333703112 P 785d3ca35dcd4aea838abdfdf3584460: stopping tablet replica
I20260928 02:34:57.593499 8441 raft_consensus.cc:2229] T 62f5d331889e4feda04688c333703112 P 785d3ca35dcd4aea838abdfdf3584460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:34:57.593532 8441 raft_consensus.cc:2258] T 62f5d331889e4feda04688c333703112 P 785d3ca35dcd4aea838abdfdf3584460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:57.593631 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 7977
I20260928 02:34:57.594576 8182 tablet_service.cc:1514] Processing DeleteTablet for tablet 62f5d331889e4feda04688c333703112 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40210
I20260928 02:34:57.594684 8440 tablet_replica.cc:330] T 62f5d331889e4feda04688c333703112 P ee5ade571f0349a1b4e3fa4edbeae2e5: stopping tablet replica
I20260928 02:34:57.594736 8440 raft_consensus.cc:2229] T 62f5d331889e4feda04688c333703112 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:34:57.594775 8440 raft_consensus.cc:2258] T 62f5d331889e4feda04688c333703112 P ee5ade571f0349a1b4e3fa4edbeae2e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:57.595017 8440 ts_tablet_manager.cc:1905] T 62f5d331889e4feda04688c333703112 P ee5ade571f0349a1b4e3fa4edbeae2e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:57.595290 8322 tablet_service.cc:1514] Processing DeleteTablet for tablet 62f5d331889e4feda04688c333703112 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:34:57 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32856
I20260928 02:34:57.595371 8439 tablet_replica.cc:330] T 62f5d331889e4feda04688c333703112 P a1187d9ed1b24dac9b4887d60838afb9: stopping tablet replica
I20260928 02:34:57.595418 8439 raft_consensus.cc:2229] T 62f5d331889e4feda04688c333703112 P a1187d9ed1b24dac9b4887d60838afb9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:34:57.595482 8439 raft_consensus.cc:2258] T 62f5d331889e4feda04688c333703112 P a1187d9ed1b24dac9b4887d60838afb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:34:57.596413 8440 ts_tablet_manager.cc:1918] T 62f5d331889e4feda04688c333703112 P ee5ade571f0349a1b4e3fa4edbeae2e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:34:57.596459 8440 log.cc:1196] T 62f5d331889e4feda04688c333703112 P ee5ade571f0349a1b4e3fa4edbeae2e5: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-1/wal/wals/62f5d331889e4feda04688c333703112
I20260928 02:34:57.596633 8440 ts_tablet_manager.cc:1939] T 62f5d331889e4feda04688c333703112 P ee5ade571f0349a1b4e3fa4edbeae2e5: Deleting consensus metadata
I20260928 02:34:57.596887 7890 catalog_manager.cc:4841] TS ee5ade571f0349a1b4e3fa4edbeae2e5 (127.0.98.2:42811): tablet 62f5d331889e4feda04688c333703112 (table test-table [id=bbc1fa6c2ca649e392b1d6953a07a1cb]) successfully deleted
I20260928 02:34:57.597445 8439 ts_tablet_manager.cc:1905] T 62f5d331889e4feda04688c333703112 P a1187d9ed1b24dac9b4887d60838afb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:34:57.598443 8439 ts_tablet_manager.cc:1918] T 62f5d331889e4feda04688c333703112 P a1187d9ed1b24dac9b4887d60838afb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:34:57.598488 8439 log.cc:1196] T 62f5d331889e4feda04688c333703112 P a1187d9ed1b24dac9b4887d60838afb9: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790562862804661-392-0/minicluster-data/ts-2/wal/wals/62f5d331889e4feda04688c333703112
I20260928 02:34:57.598654 8439 ts_tablet_manager.cc:1939] T 62f5d331889e4feda04688c333703112 P a1187d9ed1b24dac9b4887d60838afb9: Deleting consensus metadata
I20260928 02:34:57.598954 7889 catalog_manager.cc:4841] TS a1187d9ed1b24dac9b4887d60838afb9 (127.0.98.3:35633): tablet 62f5d331889e4feda04688c333703112 (table test-table [id=bbc1fa6c2ca649e392b1d6953a07a1cb]) successfully deleted
W20260928 02:34:57.601128 7889 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43567 (error 108)
I20260928 02:34:57.601377 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 8117
I20260928 02:34:57.601925 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:57.601440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43567 (local address 127.0.0.1:47704)
0928 02:34:57.601849 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0928 02:34:57.601888 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":36,"threads_started":1}
W20260928 02:34:57.601994 7889 catalog_manager.cc:4584] TS 785d3ca35dcd4aea838abdfdf3584460 (127.0.98.1:43567): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 62f5d331889e4feda04688c333703112: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43567: connect: Connection refused (error 111)
I20260928 02:34:57.606173 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 8257
I20260928 02:34:57.610038 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 7873
2026-09-28T02:34:57Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2441 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:34:57 dist-test-slave-fz47 krb5kdc[8502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:34:57 dist-test-slave-fz47 krb5kdc[8502](info): set up 2 sockets
Sep 28 02:34:57 dist-test-slave-fz47 krb5kdc[8502](info): commencing operation
krb5kdc: starting...
W20260928 02:34:58.545475 392 mini_kdc.cc:121] Time spent starting KDC: real 0.917s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:34:58 dist-test-slave-fz47 krb5kdc[8502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:34:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:34:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 02:34:58.656098 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40421
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:58.734273 8515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:58.734452 8515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:58.734472 8515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:58.735574 8515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:34:58.735610 8515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:58.735622 8515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:58.735634 8515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:34:58.735644 8515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:34:58.737071 8515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:58.737295 8515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:58.737494 8515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:58.742147 8521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:58.742203 8520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:58.742230 8523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:58.742264 8515 server_base.cc:1034] running on GCE node
I20260928 02:34:58.742590 8515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:58.742858 8515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:58.744000 8515 hybrid_clock.cc:648] HybridClock initialized: now 1790562898743989 us; error 24 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T02:34:58Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1144 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:34:58 dist-test-slave-fz47 krb5kdc[8528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:34:58 dist-test-slave-fz47 krb5kdc[8528](info): set up 2 sockets
Sep 28 02:34:58 dist-test-slave-fz47 krb5kdc[8528](info): commencing operation
krb5kdc: starting...
W20260928 02:34:59.689445 392 mini_kdc.cc:121] Time spent starting KDC: real 0.917s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:34:59 dist-test-slave-fz47 krb5kdc[8528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:34:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:34:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 02:34:59.800891 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34415
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:34:59.878435 8541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:34:59.878643 8541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:34:59.878671 8541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:34:59.879781 8541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:34:59.879818 8541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:34:59.879835 8541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:34:59.879850 8541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:34:59.879868 8541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:34:59.881546 8541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:34:59.881783 8541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:34:59.881968 8541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:34:59.886462 8549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:59.886511 8541 server_base.cc:1034] running on GCE node
W20260928 02:34:59.886494 8547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:34:59.886469 8546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:34:59.886932 8541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:34:59.887208 8541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:34:59.888334 8541 hybrid_clock.cc:648] HybridClock initialized: now 1790562899888326 us; error 27 us; skew 500 ppm
Sep 28 02:34:59 dist-test-slave-fz47 krb5kdc[8528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:34:59.890574 8541 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:34:59.890959 8541 webserver.cc:460] Webserver started at http://127.0.98.62:45027/ using document root <none> and password file <none>
I20260928 02:34:59.891160 8541 fs_manager.cc:359] Metadata directory not provided
I20260928 02:34:59.891201 8541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:34:59.891314 8541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:34:59.892333 8541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "58b4e62c3f824bf79afd81eb529315cb"
format_stamp: "Formatted at 2026-09-28 02:34:59 on dist-test-slave-fz47"
server_key: "6bfeab533c799e06b2d5e0ee269dbb0e"
server_key_iv: "d43f351dd4a660cd13ba47902da8ac04"
server_key_version: "encryptionkey@0"
I20260928 02:34:59.892645 8541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "58b4e62c3f824bf79afd81eb529315cb"
format_stamp: "Formatted at 2026-09-28 02:34:59 on dist-test-slave-fz47"
server_key: "6bfeab533c799e06b2d5e0ee269dbb0e"
server_key_iv: "d43f351dd4a660cd13ba47902da8ac04"
server_key_version: "encryptionkey@0"
I20260928 02:34:59.893919 8541 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:34:59.894582 8556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:34:59.894730 8541 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:34:59.894807 8541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "58b4e62c3f824bf79afd81eb529315cb"
format_stamp: "Formatted at 2026-09-28 02:34:59 on dist-test-slave-fz47"
server_key: "6bfeab533c799e06b2d5e0ee269dbb0e"
server_key_iv: "d43f351dd4a660cd13ba47902da8ac04"
server_key_version: "encryptionkey@0"
I20260928 02:34:59.894863 8541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:34:59.915795 8541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:34:59.917506 8541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:34:59.917640 8541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:34:59.921078 8541 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34415
I20260928 02:34:59.921119 8608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34415 every 8 connection(s)
I20260928 02:34:59.921593 8541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:34:59.922199 8609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:34:59.924297 8609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb: Bootstrap starting.
I20260928 02:34:59.924836 8609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb: Neither blocks nor log segments found. Creating new log.
I20260928 02:34:59.924866 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 8541
I20260928 02:34:59.924986 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:34:59.925159 392 external_mini_cluster.cc:1428] Setting key 41d481791653b42c98ffcac40cb79124
I20260928 02:34:59.925154 8609 log.cc:824] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb: Log is configured to *not* fsync() on all Append() calls
I20260928 02:34:59.925886 8609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb: No bootstrap required, opened a new log
I20260928 02:34:59.927857 8609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58b4e62c3f824bf79afd81eb529315cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34415 } }
I20260928 02:34:59.927986 8609 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:34:59.928017 8609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58b4e62c3f824bf79afd81eb529315cb, State: Initialized, Role: FOLLOWER
I20260928 02:34:59.928133 8609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58b4e62c3f824bf79afd81eb529315cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34415 } }
I20260928 02:34:59.928203 8609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:34:59.928233 8609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:34:59.928277 8609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:34:59.928758 8609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58b4e62c3f824bf79afd81eb529315cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34415 } }
I20260928 02:34:59.928848 8609 leader_election.cc:302] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58b4e62c3f824bf79afd81eb529315cb; no voters:
I20260928 02:34:59.928992 8609 leader_election.cc:288] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:34:59.929032 8614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:34:59.929149 8614 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [term 1 LEADER]: Becoming Leader. State: Replica: 58b4e62c3f824bf79afd81eb529315cb, State: Running, Role: LEADER
I20260928 02:34:59.929221 8609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:34:59.929224 8614 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58b4e62c3f824bf79afd81eb529315cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34415 } }
I20260928 02:34:59.929633 8616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58b4e62c3f824bf79afd81eb529315cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58b4e62c3f824bf79afd81eb529315cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34415 } } }
I20260928 02:34:59.929983 8616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [sys.catalog]: This master's current role is: LEADER
I20260928 02:34:59.929572 8614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58b4e62c3f824bf79afd81eb529315cb. Latest consensus state: current_term: 1 leader_uuid: "58b4e62c3f824bf79afd81eb529315cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58b4e62c3f824bf79afd81eb529315cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34415 } } }
I20260928 02:34:59.930055 8614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [sys.catalog]: This master's current role is: LEADER
Sep 28 02:34:59 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:34:59.931687 8628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:34:59.932319 8628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:34:59.933871 8628 catalog_manager.cc:1347] Generated new cluster ID: bb1e1c3a22414dbe81dabea724773b85
I20260928 02:34:59.933916 8628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:34:59.935169 8613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:34:59.925632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34892 (local address 127.0.98.62:34415)
0928 02:34:59.925783 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 02:34:59.925793 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:34:59.925827 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:34:59.926290 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:34:59.926297 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:34:59.926342 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:34:59.926460 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:34:59.927026 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:59.927893 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:59.928712 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:34:59.929274 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:34:59.929651 (+ 377us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:34:59.931872 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:34:59.931877 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:34:59.931877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:34:59.931888 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:34:59.933595 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:59.933972 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:59.933975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:59.933975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:59.934074 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:34:59.934276 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:34:59.934277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:34:59.934277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:34:59.934576 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:34:59.934728 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 02:34:59.934830 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 02:34:59.934968 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
I20260928 02:34:59.947803 8628 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:34:59.948616 8628 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:34:59.951956 8628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb: Generated new TSK 0
I20260928 02:34:59.952134 8628 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 02:34:59.982069 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34415
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:35:00.072816 8635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:35:00.073035 8635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:35:00.073065 8635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:35:00.074236 8635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:35:00.074282 8635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:35:00.074316 8635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:35:00.075775 8635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:35:00.076001 8635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:35:00.076206 8635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:35:00.080878 8643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:35:00.080861 8641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:35:00.080857 8640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:35:00.081037 8635 server_base.cc:1034] running on GCE node
I20260928 02:35:00.081243 8635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:35:00.081488 8635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:35:00.082633 8635 hybrid_clock.cc:648] HybridClock initialized: now 1790562900082620 us; error 29 us; skew 500 ppm
Sep 28 02:35:00 dist-test-slave-fz47 krb5kdc[8528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:35:00.084889 8635 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 02:35:00.085311 8635 webserver.cc:460] Webserver started at http://127.0.98.1:38875/ using document root <none> and password file <none>
I20260928 02:35:00.085561 8635 fs_manager.cc:359] Metadata directory not provided
I20260928 02:35:00.085613 8635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:35:00.085740 8635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:35:00.086812 8635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-0/data/instance:
uuid: "8ae5056c44ac45e6870881b085d93952"
format_stamp: "Formatted at 2026-09-28 02:35:00 on dist-test-slave-fz47"
server_key: "dd8904c26835172de5a72c93553d2603"
server_key_iv: "27aaeb2042fe8fa1be88d44b8acb253c"
server_key_version: "encryptionkey@0"
I20260928 02:35:00.087118 8635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8ae5056c44ac45e6870881b085d93952"
format_stamp: "Formatted at 2026-09-28 02:35:00 on dist-test-slave-fz47"
server_key: "dd8904c26835172de5a72c93553d2603"
server_key_iv: "27aaeb2042fe8fa1be88d44b8acb253c"
server_key_version: "encryptionkey@0"
I20260928 02:35:00.088344 8635 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:35:00.089054 8650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:00.089217 8635 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:35:00.089318 8635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "8ae5056c44ac45e6870881b085d93952"
format_stamp: "Formatted at 2026-09-28 02:35:00 on dist-test-slave-fz47"
server_key: "dd8904c26835172de5a72c93553d2603"
server_key_iv: "27aaeb2042fe8fa1be88d44b8acb253c"
server_key_version: "encryptionkey@0"
I20260928 02:35:00.089377 8635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:35:00.100363 8635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:35:00.102402 8635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:35:00.102532 8635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:35:00.102751 8635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:35:00.103029 8635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:35:00.103060 8635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:00.103088 8635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:35:00.103104 8635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:00.108873 8635 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34887
I20260928 02:35:00.108924 8763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34887 every 8 connection(s)
I20260928 02:35:00.109231 8635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 02:35:00 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:35:00.116705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 8635
I20260928 02:35:00.116791 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-0/wal/instance
I20260928 02:35:00.116966 392 external_mini_cluster.cc:1428] Setting key f7a32ee8421f3d07cf8d06b97f170c29
I20260928 02:35:00.118026 8613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:00.109995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48155 (local address 127.0.98.62:34415)
0928 02:35:00.110038 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:00.110040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:00.110217 (+ 177us) server_negotiation.cc:407] Connection header received
0928 02:35:00.111123 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:00.111124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:00.111134 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:00.111185 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:00.111748 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.112257 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.113171 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.113397 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.113447 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:00.115362 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:35:00.115363 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:35:00.115364 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:35:00.115371 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:35:00.116638 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:35:00.117040 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:35:00.117041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:35:00.117041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:35:00.117126 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:35:00.117404 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:35:00.117405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:35:00.117405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:35:00.117517 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:35:00.117576 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 02:35:00.117852 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 02:35:00.117934 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:35:00.118151 8766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:00.109964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.1:48155)
0928 02:35:00.110111 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:00.110145 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:00.111029 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:00.111240 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:00.111243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:00.111478 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:00.111675 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:00.111680 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.112319 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.112319 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:00.113109 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:00.113111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.113455 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.113455 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:00.113561 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:00.113997 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:00.114005 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:00.115278 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:35:00.116712 (+ 1434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:35:00.116718 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:35:00.116721 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:35:00.116967 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:35:00.117195 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:35:00.117196 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:35:00.117196 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:35:00.117350 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:35:00.117573 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:35:00.117578 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:35:00.117731 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 02:35:00.117814 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 02:35:00.117931 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
I20260928 02:35:00.118624 8764 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34415
I20260928 02:35:00.118706 8764 heartbeater.cc:464] Registering TS with master...
I20260928 02:35:00.118871 8764 heartbeater.cc:511] Master 127.0.98.62:34415 requested a full tablet report, sending...
I20260928 02:35:00.119341 8573 ts_manager.cc:194] Registered new tserver with Master: 8ae5056c44ac45e6870881b085d93952 (127.0.98.1:34887)
I20260928 02:35:00.120088 8573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48155
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 02:35:00.139433 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34415
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:35:00.223081 8769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:35:00.223276 8769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:35:00.223299 8769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:35:00.224422 8769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:35:00.224463 8769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:35:00.224488 8769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:35:00.225967 8769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:35:00.226179 8769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:35:00.226385 8769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:35:00.231123 8769 server_base.cc:1034] running on GCE node
W20260928 02:35:00.231113 8774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:35:00.231122 8775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:35:00.232472 8777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:35:00.233052 8769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:35:00.233381 8769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:35:00.234575 8769 hybrid_clock.cc:648] HybridClock initialized: now 1790562900234537 us; error 51 us; skew 500 ppm
Sep 28 02:35:00 dist-test-slave-fz47 krb5kdc[8528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:35:00.238529 8769 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 02:35:00.239261 8769 webserver.cc:460] Webserver started at http://127.0.98.2:46037/ using document root <none> and password file <none>
I20260928 02:35:00.239753 8769 fs_manager.cc:359] Metadata directory not provided
I20260928 02:35:00.239815 8769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:35:00.239914 8769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:35:00.240963 8769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-1/data/instance:
uuid: "5696d2c69e924f2c83cda3923e64416d"
format_stamp: "Formatted at 2026-09-28 02:35:00 on dist-test-slave-fz47"
server_key: "07d277eab09fe6401d018b77769347d6"
server_key_iv: "9ad684de3425f448e04b70794714c39b"
server_key_version: "encryptionkey@0"
I20260928 02:35:00.241295 8769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5696d2c69e924f2c83cda3923e64416d"
format_stamp: "Formatted at 2026-09-28 02:35:00 on dist-test-slave-fz47"
server_key: "07d277eab09fe6401d018b77769347d6"
server_key_iv: "9ad684de3425f448e04b70794714c39b"
server_key_version: "encryptionkey@0"
I20260928 02:35:00.242837 8769 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:35:00.243697 8784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:00.244130 8769 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:35:00.244671 8769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "5696d2c69e924f2c83cda3923e64416d"
format_stamp: "Formatted at 2026-09-28 02:35:00 on dist-test-slave-fz47"
server_key: "07d277eab09fe6401d018b77769347d6"
server_key_iv: "9ad684de3425f448e04b70794714c39b"
server_key_version: "encryptionkey@0"
I20260928 02:35:00.244768 8769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:35:00.271018 8769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:35:00.273433 8769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:35:00.273588 8769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:35:00.273813 8769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:35:00.274122 8769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:35:00.274156 8769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:00.274187 8769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:35:00.274209 8769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:00.279593 8769 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42403
I20260928 02:35:00.279623 8897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42403 every 8 connection(s)
I20260928 02:35:00.279929 8769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
I20260928 02:35:00.284111 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 8769
I20260928 02:35:00.284217 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-1/wal/instance
I20260928 02:35:00.284407 392 external_mini_cluster.cc:1428] Setting key 2df85dc09ab5cc6a372ba15d5cb96dfc
Sep 28 02:35:00 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:35:00.289438 8613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:00.280726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53161 (local address 127.0.98.62:34415)
0928 02:35:00.280809 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:00.280811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:00.280934 (+ 123us) server_negotiation.cc:407] Connection header received
0928 02:35:00.281969 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:00.281971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:00.281981 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:00.282038 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:00.282662 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.283159 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.284288 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.284500 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.284541 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:00.286835 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:35:00.286836 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:35:00.286837 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:35:00.286845 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:35:00.288172 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:35:00.288540 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:35:00.288541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:35:00.288542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:35:00.288598 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:35:00.288817 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:35:00.288817 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:35:00.288818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:35:00.288930 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:35:00.288986 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:35:00.289247 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 02:35:00.289348 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:35:00.289626 8900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:00.280644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.2:53161)
0928 02:35:00.280791 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:00.280828 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:00.281836 (+ 1008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:00.282098 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:00.282101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:00.282359 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:00.282557 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:00.282561 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.283267 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.283269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:00.284182 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:00.284186 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.284613 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.284613 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:00.284730 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:00.285319 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:00.285329 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:00.286752 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:35:00.288249 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:35:00.288256 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:35:00.288258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:35:00.288475 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:35:00.288651 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:35:00.288652 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:35:00.288652 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:35:00.288762 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:35:00.288983 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:35:00.288990 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:35:00.289131 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 02:35:00.289219 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 02:35:00.289396 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
I20260928 02:35:00.290050 8898 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34415
I20260928 02:35:00.290133 8898 heartbeater.cc:464] Registering TS with master...
I20260928 02:35:00.290365 8898 heartbeater.cc:511] Master 127.0.98.62:34415 requested a full tablet report, sending...
I20260928 02:35:00.290833 8573 ts_manager.cc:194] Registered new tserver with Master: 5696d2c69e924f2c83cda3923e64416d (127.0.98.2:42403)
I20260928 02:35:00.291203 8573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53161
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 02:35:00.308298 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34415
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:35:00.388729 8903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:35:00.388933 8903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:35:00.388963 8903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:35:00.390466 8903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:35:00.390519 8903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:35:00.390556 8903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:35:00.392623 8903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:35:00.392858 8903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:35:00.393095 8903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:35:00.399027 8909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:35:00.399024 8911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:35:00.399065 8908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:35:00.399333 8903 server_base.cc:1034] running on GCE node
I20260928 02:35:00.399523 8903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:35:00.399740 8903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:35:00.400885 8903 hybrid_clock.cc:648] HybridClock initialized: now 1790562900400869 us; error 28 us; skew 500 ppm
Sep 28 02:35:00 dist-test-slave-fz47 krb5kdc[8528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:35:00.403235 8903 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 02:35:00.403677 8903 webserver.cc:460] Webserver started at http://127.0.98.3:40509/ using document root <none> and password file <none>
I20260928 02:35:00.403889 8903 fs_manager.cc:359] Metadata directory not provided
I20260928 02:35:00.403931 8903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:35:00.404053 8903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:35:00.405045 8903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-2/data/instance:
uuid: "4a6a5c3915a24905b78537f2e803f58e"
format_stamp: "Formatted at 2026-09-28 02:35:00 on dist-test-slave-fz47"
server_key: "59aa75ed25ae843fcef58d57a7e40033"
server_key_iv: "b12265f38d2ffd0810cd58faca682f35"
server_key_version: "encryptionkey@0"
I20260928 02:35:00.405398 8903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4a6a5c3915a24905b78537f2e803f58e"
format_stamp: "Formatted at 2026-09-28 02:35:00 on dist-test-slave-fz47"
server_key: "59aa75ed25ae843fcef58d57a7e40033"
server_key_iv: "b12265f38d2ffd0810cd58faca682f35"
server_key_version: "encryptionkey@0"
I20260928 02:35:00.406579 8903 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:35:00.407227 8918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:00.407379 8903 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:35:00.407444 8903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "4a6a5c3915a24905b78537f2e803f58e"
format_stamp: "Formatted at 2026-09-28 02:35:00 on dist-test-slave-fz47"
server_key: "59aa75ed25ae843fcef58d57a7e40033"
server_key_iv: "b12265f38d2ffd0810cd58faca682f35"
server_key_version: "encryptionkey@0"
I20260928 02:35:00.407509 8903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:35:00.418637 8903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:35:00.420634 8903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:35:00.420754 8903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:35:00.420953 8903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:35:00.421222 8903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:35:00.421281 8903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:00.421311 8903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:35:00.421335 8903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:00.426710 8903 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34145
I20260928 02:35:00.426764 9031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34145 every 8 connection(s)
I20260928 02:35:00.427034 8903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
I20260928 02:35:00.432550 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 8903
I20260928 02:35:00.432651 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-2/wal/instance
I20260928 02:35:00.432857 392 external_mini_cluster.cc:1428] Setting key 73805fc70f84ae15e4dfa77d8dce2a19
Sep 28 02:35:00 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:35:00.436126 8613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:00.427651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50305 (local address 127.0.98.62:34415)
0928 02:35:00.427705 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:00.427707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:00.427906 (+ 199us) server_negotiation.cc:407] Connection header received
0928 02:35:00.428804 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:00.428805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:00.428817 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:00.428886 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:00.429551 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.430089 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.431180 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.431369 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.431419 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:00.433713 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:35:00.433714 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:35:00.433715 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:35:00.433722 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:35:00.434923 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:35:00.435285 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:35:00.435286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:35:00.435286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:35:00.435340 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:35:00.435549 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:35:00.435550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:35:00.435550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:35:00.435658 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:35:00.435712 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:35:00.435964 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 02:35:00.436039 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 02:35:00.436300 9034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:00.427652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.3:50305)
0928 02:35:00.427795 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:00.427829 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:00.428704 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:00.428999 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:00.429003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:00.429234 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:00.429452 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:00.429465 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.430180 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.430181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:00.431120 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:00.431122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.431425 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.431426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:00.431547 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:00.432072 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:00.432080 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:00.433620 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:35:00.434995 (+ 1375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:35:00.435001 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:35:00.435003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:35:00.435221 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:35:00.435392 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:35:00.435392 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:35:00.435393 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:35:00.435496 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:35:00.435710 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:35:00.435716 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:35:00.435846 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 02:35:00.435943 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 02:35:00.436095 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
I20260928 02:35:00.436741 9032 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34415
I20260928 02:35:00.436826 9032 heartbeater.cc:464] Registering TS with master...
I20260928 02:35:00.436997 9032 heartbeater.cc:511] Master 127.0.98.62:34415 requested a full tablet report, sending...
I20260928 02:35:00.437333 8573 ts_manager.cc:194] Registered new tserver with Master: 4a6a5c3915a24905b78537f2e803f58e (127.0.98.3:34145)
I20260928 02:35:00.437665 8573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50305
I20260928 02:35:00.444620 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260928 02:35:00.503438 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 8635
I20260928 02:35:00.507230 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 8769
I20260928 02:35:00.510952 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 8903
I20260928 02:35:00.515089 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 8541
I20260928 02:35:00.519483 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34415
--webserver_interface=127.0.98.62
--webserver_port=45027
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:35:00.597059 9041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:35:00.597249 9041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:35:00.597286 9041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:35:00.598389 9041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:35:00.598430 9041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:35:00.598443 9041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:35:00.598454 9041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:35:00.598464 9041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:35:00.599915 9041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:35:00.600122 9041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:35:00.600319 9041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:35:00.604816 9046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:35:00.604841 9049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:35:00.604867 9041 server_base.cc:1034] running on GCE node
W20260928 02:35:00.604816 9047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:35:00.605327 9041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:35:00.605602 9041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:35:00.606737 9041 hybrid_clock.cc:648] HybridClock initialized: now 1790562900606727 us; error 30 us; skew 500 ppm
Sep 28 02:35:00 dist-test-slave-fz47 krb5kdc[8528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:35:00.609122 9041 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:35:00.609551 9041 webserver.cc:460] Webserver started at http://127.0.98.62:45027/ using document root <none> and password file <none>
I20260928 02:35:00.609760 9041 fs_manager.cc:359] Metadata directory not provided
I20260928 02:35:00.609817 9041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:35:00.611074 9041 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:35:00.611685 9056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:00.611828 9041 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:35:00.611905 9041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "58b4e62c3f824bf79afd81eb529315cb"
format_stamp: "Formatted at 2026-09-28 02:34:59 on dist-test-slave-fz47"
server_key: "6bfeab533c799e06b2d5e0ee269dbb0e"
server_key_iv: "d43f351dd4a660cd13ba47902da8ac04"
server_key_version: "encryptionkey@0"
I20260928 02:35:00.612227 9041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:35:00.625597 9041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:35:00.627621 9041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:35:00.627750 9041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:35:00.631345 9041 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34415
I20260928 02:35:00.631389 9108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34415 every 8 connection(s)
I20260928 02:35:00.631855 9041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:35:00.633010 9109 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:35:00.633442 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 9041
I20260928 02:35:00.633720 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34887
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34415
--webserver_port=38875
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:35:00.633739 9109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb: Bootstrap starting.
I20260928 02:35:00.641626 9109 log.cc:824] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb: Log is configured to *not* fsync() on all Append() calls
I20260928 02:35:00.643323 9109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:35:00.643570 9109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb: Bootstrap complete.
I20260928 02:35:00.645586 9109 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58b4e62c3f824bf79afd81eb529315cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34415 } }
I20260928 02:35:00.645803 9109 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58b4e62c3f824bf79afd81eb529315cb, State: Initialized, Role: FOLLOWER
I20260928 02:35:00.645955 9109 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58b4e62c3f824bf79afd81eb529315cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34415 } }
I20260928 02:35:00.646018 9109 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:35:00.646059 9109 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:35:00.646102 9109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:35:00.647039 9109 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58b4e62c3f824bf79afd81eb529315cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34415 } }
I20260928 02:35:00.647122 9109 leader_election.cc:302] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58b4e62c3f824bf79afd81eb529315cb; no voters:
I20260928 02:35:00.647280 9109 leader_election.cc:288] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:35:00.647330 9113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:35:00.647442 9113 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [term 2 LEADER]: Becoming Leader. State: Replica: 58b4e62c3f824bf79afd81eb529315cb, State: Running, Role: LEADER
I20260928 02:35:00.647535 9109 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:35:00.647522 9113 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58b4e62c3f824bf79afd81eb529315cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34415 } }
I20260928 02:35:00.647827 9113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58b4e62c3f824bf79afd81eb529315cb. Latest consensus state: current_term: 2 leader_uuid: "58b4e62c3f824bf79afd81eb529315cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58b4e62c3f824bf79afd81eb529315cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34415 } } }
I20260928 02:35:00.647895 9113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [sys.catalog]: This master's current role is: LEADER
I20260928 02:35:00.648032 9115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "58b4e62c3f824bf79afd81eb529315cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58b4e62c3f824bf79afd81eb529315cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34415 } } }
I20260928 02:35:00.648089 9115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb [sys.catalog]: This master's current role is: LEADER
I20260928 02:35:00.648193 9118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:35:00.648662 9118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:35:00.648785 9118 catalog_manager.cc:1259] Loaded cluster ID: bb1e1c3a22414dbe81dabea724773b85
I20260928 02:35:00.648821 9118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:35:00.649988 9118 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:35:00.650158 9118 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 58b4e62c3f824bf79afd81eb529315cb: Loaded TSK: 0
I20260928 02:35:00.650365 9118 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:35:00.713768 9111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:35:00.713970 9111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:35:00.713999 9111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:35:00.715265 9111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:35:00.715308 9111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:35:00.715344 9111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:35:00.716825 9111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38875
--tserver_master_addrs=127.0.98.62:34415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:35:00.717042 9111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:35:00.717244 9111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:35:00.721972 9136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:35:00.721987 9135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:35:00.722123 9138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:35:00.722232 9111 server_base.cc:1034] running on GCE node
I20260928 02:35:00.722470 9111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:35:00.722702 9111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:35:00.723845 9111 hybrid_clock.cc:648] HybridClock initialized: now 1790562900723826 us; error 35 us; skew 500 ppm
Sep 28 02:35:00 dist-test-slave-fz47 krb5kdc[8528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562900, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:35:00.726373 9111 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 02:35:00.726752 9111 webserver.cc:460] Webserver started at http://127.0.98.1:38875/ using document root <none> and password file <none>
I20260928 02:35:00.726936 9111 fs_manager.cc:359] Metadata directory not provided
I20260928 02:35:00.726994 9111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:35:00.728340 9111 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:35:00.728926 9145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:00.729085 9111 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:35:00.729162 9111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "8ae5056c44ac45e6870881b085d93952"
format_stamp: "Formatted at 2026-09-28 02:35:00 on dist-test-slave-fz47"
server_key: "dd8904c26835172de5a72c93553d2603"
server_key_iv: "27aaeb2042fe8fa1be88d44b8acb253c"
server_key_version: "encryptionkey@0"
I20260928 02:35:00.729512 9111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:35:00.742630 9111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:35:00.744567 9111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:35:00.744693 9111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:35:00.744891 9111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:35:00.745158 9111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:35:00.745190 9111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:00.745217 9111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:35:00.745241 9111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:00.751197 9111 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34887
I20260928 02:35:00.751295 9258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34887 every 8 connection(s)
I20260928 02:35:00.751557 9111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
I20260928 02:35:00.758217 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 9111
I20260928 02:35:00.758500 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42403
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34415
--webserver_port=46037
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:35:00 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:00 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:00.760070 9261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:00.752247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.1:52071)
0928 02:35:00.752395 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:00.752434 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:00.752864 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:00.753325 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:00.753329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:00.753852 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:00.754056 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:00.754060 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.755703 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.755704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:00.756466 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:00.756468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.757213 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.757213 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:00.757362 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:00.757825 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:00.757833 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:00.759884 (+ 2051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
W20260928 02:35:00.760250 9259 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34415 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:34415: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
W20260928 02:35:00.760592 9262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:00.752320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52071 (local address 127.0.98.62:34415)
0928 02:35:00.752510 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:00.752518 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:00.752556 (+ 38us) server_negotiation.cc:407] Connection header received
0928 02:35:00.753052 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:00.753059 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:00.753115 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:00.753305 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:00.754188 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.755618 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.756548 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.757138 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.757231 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:00.760037 (+ 2806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52071: BlockingRecv error: recv got EOF from 127.0.98.1:52071 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":83,"threads_started":1}
Sep 28 02:35:00 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:00 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:00.765769 9261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:00.760472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.1:36489)
0928 02:35:00.760784 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:00.760794 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:00.760848 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:00.760989 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:00.760990 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:00.761116 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:00.761160 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:00.761162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.762060 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.762062 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:00.762831 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:00.762834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.763244 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.763245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:00.763311 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:00.763730 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:00.763735 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:00.765711 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":295}
W20260928 02:35:00.765869 9262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:00.760523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36489 (local address 127.0.98.62:34415)
0928 02:35:00.760717 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:00.760719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:00.760851 (+ 132us) server_negotiation.cc:407] Connection header received
0928 02:35:00.760894 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:00.760895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:00.760912 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:00.760968 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:00.761225 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.761945 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.762924 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.763170 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.763237 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:00.765811 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36489: BlockingRecv error: recv got EOF from 127.0.98.1:36489 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":96,"threads_started":1}
Sep 28 02:35:00 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:00 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:00.770949 9261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:00.766176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.1:47581)
0928 02:35:00.766290 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:00.766299 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:00.766342 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:00.766472 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:00.766473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:00.766599 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:00.766631 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:00.766632 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.767468 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.767469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:00.768258 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:00.768259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.768691 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.768692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:00.768740 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:00.769139 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:00.769144 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:00.770873 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
W20260928 02:35:00.771010 9262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:00.766217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47581 (local address 127.0.98.62:34415)
0928 02:35:00.766250 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:00.766251 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:00.766348 (+ 97us) server_negotiation.cc:407] Connection header received
0928 02:35:00.766389 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:00.766390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:00.766401 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:00.766462 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:00.766701 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.767389 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.768334 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.768626 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.768691 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:00.770955 (+ 2264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47581: BlockingRecv error: recv got EOF from 127.0.98.1:47581 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 02:35:00.771255 9259 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:35:00.838177 9263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:35:00.838359 9263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:35:00.838380 9263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:35:00.839449 9263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:35:00.839481 9263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:35:00.839505 9263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:35:00.840826 9263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46037
--tserver_master_addrs=127.0.98.62:34415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:35:00.840997 9263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:35:00.841157 9263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:35:00.845892 9271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:35:00.845932 9273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:35:00.845927 9270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:35:00.846043 9263 server_base.cc:1034] running on GCE node
I20260928 02:35:00.846285 9263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:35:00.846484 9263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:35:00.847615 9263 hybrid_clock.cc:648] HybridClock initialized: now 1790562900847598 us; error 35 us; skew 500 ppm
Sep 28 02:35:00 dist-test-slave-fz47 krb5kdc[8528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562900, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:35:00.849931 9263 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 02:35:00.850330 9263 webserver.cc:460] Webserver started at http://127.0.98.2:46037/ using document root <none> and password file <none>
I20260928 02:35:00.850533 9263 fs_manager.cc:359] Metadata directory not provided
I20260928 02:35:00.850601 9263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:35:00.851773 9263 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:35:00.852280 9280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:00.852429 9263 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:00.852502 9263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "5696d2c69e924f2c83cda3923e64416d"
format_stamp: "Formatted at 2026-09-28 02:35:00 on dist-test-slave-fz47"
server_key: "07d277eab09fe6401d018b77769347d6"
server_key_iv: "9ad684de3425f448e04b70794714c39b"
server_key_version: "encryptionkey@0"
I20260928 02:35:00.852799 9263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:35:00.900082 9263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:35:00.902211 9263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:35:00.902345 9263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:35:00.902570 9263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:35:00.902853 9263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:35:00.902884 9263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:00.902912 9263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:35:00.902930 9263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:00.908138 9263 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42403
I20260928 02:35:00.908193 9393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42403 every 8 connection(s)
I20260928 02:35:00.908490 9263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
I20260928 02:35:00.913895 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 9263
Sep 28 02:35:00 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:00.914139 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34145
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34415
--webserver_port=40509
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:35:00 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:35:00.915462 9262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:00.909113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44889 (local address 127.0.98.62:34415)
0928 02:35:00.909165 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:00.909167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:00.909385 (+ 218us) server_negotiation.cc:407] Connection header received
0928 02:35:00.909881 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:00.909882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:00.909897 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:00.909943 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:00.910887 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.911365 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.912318 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.912496 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.912549 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:00.915401 (+ 2852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44889: BlockingRecv error: recv got EOF from 127.0.98.2:44889 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 02:35:00.915546 9396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:00.909110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.2:44889)
0928 02:35:00.909271 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:00.909305 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:00.909785 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:00.910025 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:00.910029 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:00.910607 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:00.910793 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:00.910798 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.911442 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.911443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:00.912236 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:00.912238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.912560 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.912561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:00.912683 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:00.913086 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:00.913094 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:00.915304 (+ 2210us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":73,"threads_started":1}
W20260928 02:35:00.915750 9394 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34415 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:34415: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 28 02:35:00 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:00 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:00.920363 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:00.915931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.2:44667)
0928 02:35:00.916042 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:00.916052 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:00.916108 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:00.916234 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:00.916234 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:00.916346 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:00.916385 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:00.916386 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.917216 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.917217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:00.917789 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:00.917792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.918224 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.918225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:00.918264 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:00.918693 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:00.918697 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:00.920293 (+ 1596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":60,"threads_started":1}
W20260928 02:35:00.920451 9262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:00.915985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44667 (local address 127.0.98.62:34415)
0928 02:35:00.916025 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:00.916027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:00.916108 (+ 81us) server_negotiation.cc:407] Connection header received
0928 02:35:00.916162 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:00.916163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:00.916173 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:00.916217 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:00.916483 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.917144 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.917863 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.918142 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.918210 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:00.920391 (+ 2181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44667: BlockingRecv error: recv got EOF from 127.0.98.2:44667 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 28 02:35:00 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:00 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:00.924713 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:00.920763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.2:48341)
0928 02:35:00.920801 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:00.920810 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:00.920844 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:00.920965 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:00.920966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:00.921085 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:00.921110 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:00.921112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.921714 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.921714 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:00.922206 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:00.922208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.922572 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.922573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:00.922627 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:00.923094 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:00.923098 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:00.924655 (+ 1557us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 02:35:00.924791 9262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:00.920825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48341 (local address 127.0.98.62:34415)
0928 02:35:00.920875 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:00.920876 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:00.920883 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:35:00.920902 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:00.920903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:00.920912 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:00.920958 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:00.921169 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.921649 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.922273 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:00.922509 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:00.922566 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:00.924739 (+ 2173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48341: BlockingRecv error: recv got EOF from 127.0.98.2:48341 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 02:35:00.924819 9394 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:35:00.993719 9397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:35:00.993908 9397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:35:00.993937 9397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:35:00.995081 9397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:35:00.995122 9397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:35:00.995157 9397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:35:00.996569 9397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34145
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40509
--tserver_master_addrs=127.0.98.62:34415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:35:00.996798 9397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:35:00.997035 9397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:35:01.001760 9404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:35:01.001771 9403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:35:01.001781 9406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:35:01.001961 9397 server_base.cc:1034] running on GCE node
I20260928 02:35:01.002151 9397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:35:01.002331 9397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:35:01.003454 9397 hybrid_clock.cc:648] HybridClock initialized: now 1790562901003443 us; error 28 us; skew 500 ppm
Sep 28 02:35:01 dist-test-slave-fz47 krb5kdc[8528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562900, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:35:01.005755 9397 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 02:35:01.006161 9397 webserver.cc:460] Webserver started at http://127.0.98.3:40509/ using document root <none> and password file <none>
I20260928 02:35:01.006332 9397 fs_manager.cc:359] Metadata directory not provided
I20260928 02:35:01.006381 9397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:35:01.007524 9397 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:35:01.007979 9413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:01.008136 9397 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:01.008210 9397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "4a6a5c3915a24905b78537f2e803f58e"
format_stamp: "Formatted at 2026-09-28 02:35:00 on dist-test-slave-fz47"
server_key: "59aa75ed25ae843fcef58d57a7e40033"
server_key_iv: "b12265f38d2ffd0810cd58faca682f35"
server_key_version: "encryptionkey@0"
I20260928 02:35:01.008512 9397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:35:01.041585 9397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:35:01.043705 9397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:35:01.043839 9397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:35:01.044044 9397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:35:01.044335 9397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:35:01.044373 9397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:01.044400 9397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:35:01.044417 9397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:01.049876 9397 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34145
I20260928 02:35:01.049939 9526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34145 every 8 connection(s)
I20260928 02:35:01.050246 9397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 02:35:01 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:01 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:35:01.057024 9262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:01.050933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41443 (local address 127.0.98.62:34415)
0928 02:35:01.050991 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:01.050993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:01.051178 (+ 185us) server_negotiation.cc:407] Connection header received
0928 02:35:01.051656 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:01.051657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:01.051669 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:01.051720 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:01.052562 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:01.053036 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:01.054112 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:01.054292 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:01.054338 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:01.056971 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41443: BlockingRecv error: recv got EOF from 127.0.98.3:41443 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 02:35:01.057037 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:01.050918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.3:41443)
0928 02:35:01.051068 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:01.051103 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:01.051565 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:01.051782 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:01.051786 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:01.052309 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:01.052491 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:01.052495 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:01.053131 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:01.053132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:01.054033 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:01.054036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:01.054374 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:01.054375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:01.054482 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:01.054934 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:01.054943 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:01.056891 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
W20260928 02:35:01.057230 9527 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34415 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:34415: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260928 02:35:01.059465 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 9397
Sep 28 02:35:01 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:01 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:01.061441 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:01.057446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.3:34427)
0928 02:35:01.057490 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:01.057500 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:01.057535 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:01.057824 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:01.057825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:01.057922 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:01.057963 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:01.057965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:01.058524 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:01.058525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:01.058997 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:01.058998 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:01.059315 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:01.059315 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:01.059376 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:01.059762 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:01.059766 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:01.061378 (+ 1612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 02:35:01.061508 9262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:01.057663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34427 (local address 127.0.98.62:34415)
0928 02:35:01.057732 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:01.057734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:01.057741 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:35:01.057758 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:01.057759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:01.057766 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:01.057816 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:01.058016 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:01.058472 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:01.059075 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:01.059250 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:01.059310 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:01.061455 (+ 2145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34427: BlockingRecv error: recv got EOF from 127.0.98.3:34427 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 02:35:01 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:01 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:01.065512 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:01.061851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.3:49857)
0928 02:35:01.061893 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:01.061903 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:01.061936 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:01.062042 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:01.062043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:01.062159 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:01.062183 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:01.062184 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:01.062739 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:01.062740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:01.063217 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:01.063219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:01.063504 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:01.063505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:01.063552 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:01.063875 (+ 323us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:01.063880 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:01.065452 (+ 1572us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 02:35:01.065620 9527 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:35:01.065649 9262 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:01.061914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49857 (local address 127.0.98.62:34415)
0928 02:35:01.061955 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:01.061957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:01.061966 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:35:01.061982 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:01.061983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:01.061991 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:01.062035 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:01.062243 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:01.062678 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:01.063280 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:01.063452 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:01.063501 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:01.065524 (+ 2023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49857: BlockingRecv error: recv got EOF from 127.0.98.3:49857 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:35:01.066466 9264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:01.059855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34906 (local address 127.0.98.62:34415)
0928 02:35:01.059928 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:01.059930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:01.060054 (+ 124us) server_negotiation.cc:407] Connection header received
0928 02:35:01.060122 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:01.060123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:01.060131 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:01.060184 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:01.060765 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:01.061308 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:01.062224 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:01.062435 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:01.062485 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:01.063570 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:35:01.063575 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:35:01.063575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:35:01.063587 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:35:01.065186 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:35:01.065515 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:35:01.065519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:35:01.065519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:35:01.065657 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:35:01.065869 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:35:01.065869 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:35:01.065870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:35:01.066083 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:35:01.066186 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 02:35:01.066277 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 02:35:01.066369 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 02:35:01 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:01 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:01.775552 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:01.771587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.1:53045)
0928 02:35:01.771706 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:01.771718 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:01.771769 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:01.771879 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:01.771880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:01.772046 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:01.772091 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:01.772093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:01.772691 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:01.772692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:01.773221 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:01.773224 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:01.773580 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:01.773581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:01.773635 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:01.774022 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:01.774026 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:01.775485 (+ 1459us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":56,"threads_started":1}
W20260928 02:35:01.775640 9532 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:01.771646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53045 (local address 127.0.98.62:34415)
0928 02:35:01.771765 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:01.771766 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:01.771775 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:35:01.771813 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:01.771813 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:01.771823 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:01.771871 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:01.772143 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:01.772633 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:01.773323 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:01.773526 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:01.773576 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:01.775575 (+ 1999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53045: BlockingRecv error: recv got EOF from 127.0.98.1:53045 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
Sep 28 02:35:01 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:01 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:01.929368 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:01.925164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.2:43735)
0928 02:35:01.925308 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:01.925325 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:01.925396 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:01.925524 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:01.925525 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:01.925677 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:01.925745 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:01.925746 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:01.926368 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:01.926368 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:01.926893 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:01.926895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:01.927207 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:01.927208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:01.927265 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:01.927652 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:01.927656 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:01.929310 (+ 1654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":65,"threads_started":1}
W20260928 02:35:01.929435 9532 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:01.925236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43735 (local address 127.0.98.62:34415)
0928 02:35:01.925326 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:01.925328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:01.925385 (+ 57us) server_negotiation.cc:407] Connection header received
0928 02:35:01.925454 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:01.925455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:01.925467 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:01.925514 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:01.925810 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:01.926301 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:01.926961 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:01.927151 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:01.927204 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:01.929383 (+ 2179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43735: BlockingRecv error: recv got EOF from 127.0.98.2:43735 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 02:35:02 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:02 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:02.070107 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:02.066000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.3:42867)
0928 02:35:02.066118 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:02.066130 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:02.066190 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:02.066323 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:02.066323 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:02.066513 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:02.066571 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:02.066574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:02.067176 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:02.067177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:02.067731 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:02.067733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:02.068011 (+ 278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:02.068012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:02.068049 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:02.068426 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:02.068430 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:02.070039 (+ 1609us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":65,"threads_started":1}
W20260928 02:35:02.070179 9532 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:02.066074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42867 (local address 127.0.98.62:34415)
0928 02:35:02.066137 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:02.066139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:02.066176 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:35:02.066238 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:02.066239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:02.066253 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:02.066311 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:02.066637 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:02.067116 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:02.067788 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:02.067958 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:02.068005 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:02.070120 (+ 2115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42867: BlockingRecv error: recv got EOF from 127.0.98.3:42867 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 02:35:02 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:02 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:02.783013 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:02.775930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.1:38729)
0928 02:35:02.776055 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:02.776065 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:02.776112 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:02.776225 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:02.776226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:02.776369 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:02.776417 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:02.776419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:02.776993 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:02.776994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:02.777546 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:02.777548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:02.777868 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:02.777869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:02.777930 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:02.781348 (+ 3418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:02.781353 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:02.782926 (+ 1573us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
W20260928 02:35:02.783061 9536 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:02.775991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38729 (local address 127.0.98.62:34415)
0928 02:35:02.776095 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:02.776097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:02.776105 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:35:02.776152 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:02.776153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:02.776162 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:02.776216 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:02.776468 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:02.776942 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:02.777608 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:02.777808 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:02.777866 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:02.783002 (+ 5136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38729: BlockingRecv error: recv got EOF from 127.0.98.1:38729 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":47,"threads_started":1}
Sep 28 02:35:02 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:02 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:02.933729 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:02.929754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.2:51939)
0928 02:35:02.929860 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:02.929869 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:02.929921 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:02.930028 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:02.930029 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:02.930168 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:02.930216 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:02.930217 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:02.930805 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:02.930806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:02.931357 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:02.931359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:02.931671 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:02.931671 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:02.931707 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:02.932102 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:02.932106 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:02.933671 (+ 1565us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":52,"threads_started":1}
W20260928 02:35:02.933815 9536 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:02.929805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51939 (local address 127.0.98.62:34415)
0928 02:35:02.929854 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:02.929856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:02.929913 (+ 57us) server_negotiation.cc:407] Connection header received
0928 02:35:02.929962 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:02.929962 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:02.929972 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:02.930019 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:02.930268 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:02.930753 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:02.931441 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:02.931621 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:02.931669 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:02.933758 (+ 2089us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51939: BlockingRecv error: recv got EOF from 127.0.98.2:51939 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 28 02:35:03 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:03 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:03.074733 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:03.070509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.3:55137)
0928 02:35:03.070643 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:03.070655 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:03.070722 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:03.070838 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:03.070839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:03.071009 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:03.071052 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:03.071054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:03.071682 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:03.071683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:03.072230 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:03.072231 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:03.072552 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:03.072553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:03.072604 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:03.073035 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:03.073039 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:03.074656 (+ 1617us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
W20260928 02:35:03.074800 9536 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:03.070624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55137 (local address 127.0.98.62:34415)
0928 02:35:03.070674 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:03.070676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:03.070703 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:35:03.070772 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:03.070773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:03.070786 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:03.070832 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:03.071116 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:03.071613 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:03.072300 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:03.072493 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:03.072548 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:03.074745 (+ 2197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55137: BlockingRecv error: recv got EOF from 127.0.98.3:55137 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 28 02:35:03 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:03 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:03.787730 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:03.783460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.1:48617)
0928 02:35:03.783577 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:03.783587 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:03.783645 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:03.783771 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:03.783772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:03.783924 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:03.783973 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:03.783974 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:03.784597 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:03.784598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:03.785131 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:03.785133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:03.785509 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:03.785510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:03.785552 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:03.785978 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:03.785983 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:03.787665 (+ 1682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
W20260928 02:35:03.787811 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:03.783550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48617 (local address 127.0.98.62:34415)
0928 02:35:03.783668 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:03.783670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:03.783677 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:35:03.783698 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:03.783698 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:03.783707 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:03.783777 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:03.784026 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:03.784520 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:03.785193 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:03.785447 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:03.785507 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:03.787756 (+ 2249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48617: BlockingRecv error: recv got EOF from 127.0.98.1:48617 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":51,"threads_started":1}
Sep 28 02:35:03 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:03 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:03.938447 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:03.934177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.2:38237)
0928 02:35:03.934302 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:03.934312 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:03.934374 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:03.934537 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:03.934538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:03.934703 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:03.934754 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:03.934756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:03.935380 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:03.935381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:03.935902 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:03.935903 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:03.936214 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:03.936215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:03.936276 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:03.936680 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:03.936684 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:03.938388 (+ 1704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
W20260928 02:35:03.938547 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:03.934260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38237 (local address 127.0.98.62:34415)
0928 02:35:03.934356 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:03.934358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:03.934370 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:35:03.934443 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:03.934444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:03.934456 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:03.934524 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:03.934826 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:03.935303 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:03.935973 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:03.936158 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:03.936211 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:03.938482 (+ 2271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38237: BlockingRecv error: recv got EOF from 127.0.98.2:38237 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 02:35:04 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:04 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:04.079325 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:04.075187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.3:55683)
0928 02:35:04.075309 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:04.075327 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:04.075381 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:04.075528 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:04.075529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:04.075681 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:04.075727 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:04.075728 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:04.076348 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:04.076349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:04.076876 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:04.076877 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:04.077179 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:04.077180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:04.077229 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:04.077627 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:04.077630 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:04.079240 (+ 1610us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":55,"threads_started":1}
W20260928 02:35:04.079380 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:04.075278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55683 (local address 127.0.98.62:34415)
0928 02:35:04.075328 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:04.075329 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:04.075410 (+ 81us) server_negotiation.cc:407] Connection header received
0928 02:35:04.075437 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:04.075438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:04.075451 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:04.075518 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:04.075787 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:04.076265 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:04.076944 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:04.077117 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:04.077158 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:04.079327 (+ 2169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55683: BlockingRecv error: recv got EOF from 127.0.98.3:55683 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 28 02:35:04 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:04 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:04.792155 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:04.788188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.1:53675)
0928 02:35:04.788297 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:04.788307 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:04.788362 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:04.788483 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:04.788483 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:04.788649 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:04.788697 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:04.788699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:04.789334 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:04.789335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:04.789868 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:04.789870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:04.790171 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:04.790171 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:04.790205 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:04.790580 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:04.790584 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:04.792098 (+ 1514us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":65,"threads_started":1}
W20260928 02:35:04.792245 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:04.788266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53675 (local address 127.0.98.62:34415)
0928 02:35:04.788382 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:04.788384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:04.788392 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:35:04.788413 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:04.788414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:04.788424 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:04.788477 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:04.788746 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:04.789239 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:04.789931 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:04.790113 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:04.790167 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:04.792179 (+ 2012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53675: BlockingRecv error: recv got EOF from 127.0.98.1:53675 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":58,"threads_started":1}
Sep 28 02:35:04 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:04 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:04.942968 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:04.938853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.2:53987)
0928 02:35:04.938962 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:04.938972 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:04.939028 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:04.939127 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:04.939128 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:04.939277 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:04.939334 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:04.939336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:04.939935 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:04.939936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:04.940534 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:04.940537 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:04.940851 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:04.940852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:04.940888 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:04.941340 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:04.941344 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:04.942910 (+ 1566us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":54,"threads_started":1}
W20260928 02:35:04.943043 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:04.938914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53987 (local address 127.0.98.62:34415)
0928 02:35:04.938981 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:04.938983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:04.939014 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:35:04.939067 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:04.939068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:04.939078 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:04.939116 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:04.939384 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:04.939882 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:04.940599 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:04.940797 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:04.940848 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:04.942987 (+ 2139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53987: BlockingRecv error: recv got EOF from 127.0.98.2:53987 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 02:35:05 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:05 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:05.083977 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:05.079805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.3:56913)
0928 02:35:05.079932 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:05.079942 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:05.080008 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:05.080132 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:05.080133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:05.080296 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:05.080342 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:05.080344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:05.081022 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:05.081024 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:05.081655 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:05.081657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:05.081971 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:05.081971 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:05.082010 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:05.082401 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:05.082405 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:05.083912 (+ 1507us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
W20260928 02:35:05.084059 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:05.079862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56913 (local address 127.0.98.62:34415)
0928 02:35:05.079928 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:05.079930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:05.080008 (+ 78us) server_negotiation.cc:407] Connection header received
0928 02:35:05.080049 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:05.080050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:05.080062 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:05.080126 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:05.080394 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:05.080951 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:05.081730 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:05.081914 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:05.081968 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:05.084000 (+ 2032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56913: BlockingRecv error: recv got EOF from 127.0.98.3:56913 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 02:35:05 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:05 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:05.797003 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:05.792607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.1:60759)
0928 02:35:05.792746 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:05.792759 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:05.792823 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:05.792950 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:05.792951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:05.793115 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:05.793160 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:05.793162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:05.793842 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:05.793843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:05.794381 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:05.794383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:05.794713 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:05.794713 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:05.794783 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:05.795202 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:05.795207 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:05.796933 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
W20260928 02:35:05.797109 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:05.792639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60759 (local address 127.0.98.62:34415)
0928 02:35:05.792755 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:05.792757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:05.792819 (+ 62us) server_negotiation.cc:407] Connection header received
0928 02:35:05.792872 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:05.792873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:05.792885 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:05.792936 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:05.793228 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:05.793770 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:05.794457 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:05.794655 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:05.794709 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:05.797022 (+ 2313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60759: BlockingRecv error: recv got EOF from 127.0.98.1:60759 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":65,"threads_started":1}
Sep 28 02:35:05 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:05 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:05.947415 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:05.943380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.2:38813)
0928 02:35:05.943501 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:05.943511 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:05.943564 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:05.943672 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:05.943673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:05.943820 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:05.943866 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:05.943868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:05.944455 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:05.944456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:05.944985 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:05.944987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:05.945343 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:05.945344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:05.945384 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:05.945781 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:05.945785 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:05.947340 (+ 1555us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":56,"threads_started":1}
W20260928 02:35:05.947484 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:05.943471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38813 (local address 127.0.98.62:34415)
0928 02:35:05.943514 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:05.943516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:05.943550 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:35:05.943603 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:05.943604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:05.943613 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:05.943661 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:05.943914 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:05.944402 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:05.945055 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:05.945280 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:05.945344 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:05.947432 (+ 2088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38813: BlockingRecv error: recv got EOF from 127.0.98.2:38813 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 28 02:35:06 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:06 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:06.088429 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:06.084398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.3:33395)
0928 02:35:06.084536 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:06.084547 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:06.084605 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:06.084709 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:06.084709 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:06.084858 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:06.084904 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:06.084905 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:06.085522 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:06.085523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:06.086057 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:06.086059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:06.086343 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:06.086344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:06.086381 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:06.086768 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:06.086772 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:06.088356 (+ 1584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":62,"threads_started":1}
W20260928 02:35:06.088503 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:06.084461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33395 (local address 127.0.98.62:34415)
0928 02:35:06.084520 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:06.084523 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:06.084599 (+ 76us) server_negotiation.cc:407] Connection header received
0928 02:35:06.084652 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:06.084653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:06.084664 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:06.084705 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:06.084949 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:06.085471 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:06.086114 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:06.086294 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:06.086335 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:06.088450 (+ 2115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33395: BlockingRecv error: recv got EOF from 127.0.98.3:33395 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 28 02:35:06 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:06 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:06.801470 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:06.797433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.1:50761)
0928 02:35:06.797550 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:06.797560 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:06.797611 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:06.797747 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:06.797747 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:06.797896 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:06.797942 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:06.797944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:06.798528 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:06.798529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:06.799055 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:06.799057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:06.799377 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:06.799377 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:06.799417 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:06.799821 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:06.799826 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:06.801412 (+ 1586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":62,"threads_started":1}
W20260928 02:35:06.801561 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:06.797489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50761 (local address 127.0.98.62:34415)
0928 02:35:06.797628 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:06.797631 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:06.797639 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:35:06.797661 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:06.797668 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:06.797679 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:06.797736 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:06.797994 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:06.798473 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:06.799125 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:06.799321 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:06.799371 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:06.801497 (+ 2126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50761: BlockingRecv error: recv got EOF from 127.0.98.1:50761 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
Sep 28 02:35:06 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:06 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:06.951763 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:06.947836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.2:52851)
0928 02:35:06.947966 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:06.947978 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:06.948037 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:06.948148 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:06.948149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:06.948300 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:06.948347 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:06.948348 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:06.948940 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:06.948941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:06.949503 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:06.949505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:06.949795 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:06.949795 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:06.949829 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:06.950210 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:06.950214 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:06.951706 (+ 1492us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
W20260928 02:35:06.951851 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:06.947895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52851 (local address 127.0.98.62:34415)
0928 02:35:06.947954 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:06.947957 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:06.948030 (+ 73us) server_negotiation.cc:407] Connection header received
0928 02:35:06.948077 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:06.948078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:06.948088 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:06.948138 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:06.948394 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:06.948863 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:06.949560 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:06.949745 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:06.949792 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:06.951793 (+ 2001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52851: BlockingRecv error: recv got EOF from 127.0.98.2:52851 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 28 02:35:07 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:07 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:07.093062 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:07.088862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.3:55041)
0928 02:35:07.088971 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:07.088983 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:07.089043 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:07.089164 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:07.089165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:07.089342 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:07.089390 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:07.089392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:07.089990 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:07.089990 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:07.090660 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:07.090662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:07.090946 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:07.090947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:07.090984 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:07.091375 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:07.091379 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:07.093005 (+ 1626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":51,"threads_started":1}
W20260928 02:35:07.093168 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:07.088936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55041 (local address 127.0.98.62:34415)
0928 02:35:07.088999 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:07.089000 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:07.089027 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:35:07.089090 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:07.089091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:07.089103 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:07.089144 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:07.089449 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:07.089928 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:07.090713 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:07.090892 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:07.090943 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:07.093090 (+ 2147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55041: BlockingRecv error: recv got EOF from 127.0.98.3:55041 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 02:35:07 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:07 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:07.813017 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:07.801978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.1:53567)
0928 02:35:07.802116 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:07.802127 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:07.802185 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:07.802329 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:07.802330 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:07.802498 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:07.802546 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:07.802547 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:07.803187 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:07.803188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:07.803721 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:07.803723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:07.804041 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:07.804042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:07.804082 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:07.811216 (+ 7134us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:07.811221 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:07.812937 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":65,"threads_started":1}
W20260928 02:35:07.813087 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:07.802017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53567 (local address 127.0.98.62:34415)
0928 02:35:07.802180 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:07.802182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:07.802191 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:35:07.802245 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:07.802246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:07.802258 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:07.802315 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:07.802602 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:07.803088 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:07.803781 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:07.803981 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:07.804040 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:07.813010 (+ 8970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53567: BlockingRecv error: recv got EOF from 127.0.98.1:53567 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":46,"threads_started":1}
Sep 28 02:35:07 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:07 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:07.956496 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:07.952197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.2:37885)
0928 02:35:07.952323 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:07.952335 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:07.952406 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:07.952527 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:07.952528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:07.952727 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:07.952773 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:07.952775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:07.953432 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:07.953433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:07.953976 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:07.953977 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:07.954308 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:07.954309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:07.954365 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:07.954777 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:07.954781 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:07.956416 (+ 1635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":66,"threads_started":1}
W20260928 02:35:07.956574 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:07.952277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37885 (local address 127.0.98.62:34415)
0928 02:35:07.952340 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:07.952342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:07.952396 (+ 54us) server_negotiation.cc:407] Connection header received
0928 02:35:07.952457 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:07.952458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:07.952469 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:07.952518 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:07.952851 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:07.953370 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:07.954051 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:07.954251 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:07.954305 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:07.956520 (+ 2215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37885: BlockingRecv error: recv got EOF from 127.0.98.2:37885 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 02:35:08 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:08 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:08.097788 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:08.093482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.3:33413)
0928 02:35:08.093643 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:08.093654 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:08.093717 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:08.093845 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:08.093845 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:08.094011 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:08.094054 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:08.094056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:08.094713 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:08.094714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:08.095234 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:08.095236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:08.095553 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:08.095554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:08.095608 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:08.096009 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:08.096012 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:08.097711 (+ 1699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":91,"threads_started":1}
W20260928 02:35:08.097878 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:08.093557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33413 (local address 127.0.98.62:34415)
0928 02:35:08.093619 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:08.093622 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:08.093710 (+ 88us) server_negotiation.cc:407] Connection header received
0928 02:35:08.093776 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:08.093777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:08.093790 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:08.093834 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:08.094138 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:08.094643 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:08.095313 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:08.095496 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:08.095550 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:08.097813 (+ 2263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33413: BlockingRecv error: recv got EOF from 127.0.98.3:33413 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 28 02:35:08 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:08 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:08.817572 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:08.813427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.1:57757)
0928 02:35:08.813555 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:08.813566 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:08.813613 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:08.813738 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:08.813739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:08.813880 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:08.813926 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:08.813928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:08.814516 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:08.814517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:08.815063 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:08.815065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:08.815371 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:08.815371 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:08.815406 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:08.815846 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:08.815850 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:08.817494 (+ 1644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":56,"threads_started":1}
W20260928 02:35:08.817647 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:08.813519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57757 (local address 127.0.98.62:34415)
0928 02:35:08.813634 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:08.813635 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:08.813642 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:35:08.813661 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:08.813661 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:08.813670 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:08.813726 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:08.813974 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:08.814458 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:08.815127 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:08.815325 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:08.815367 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:08.817588 (+ 2221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57757: BlockingRecv error: recv got EOF from 127.0.98.1:57757 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":63,"threads_started":1}
Sep 28 02:35:08 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:08 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:08.961350 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:08.956890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.2:34785)
0928 02:35:08.957064 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:08.957077 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:08.957151 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:08.957311 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:08.957312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:08.957480 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:08.957526 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:08.957528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:08.958306 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:08.958307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:08.958829 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:08.958831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:08.959155 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:08.959156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:08.959209 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:08.959629 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:08.959633 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:08.961240 (+ 1607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":82,"threads_started":1}
W20260928 02:35:08.961416 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:08.956957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34785 (local address 127.0.98.62:34415)
0928 02:35:08.957068 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:08.957071 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:08.957143 (+ 72us) server_negotiation.cc:407] Connection header received
0928 02:35:08.957212 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:08.957213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:08.957224 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:08.957302 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:08.957596 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:08.958228 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:08.958915 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:08.959098 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:08.959152 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:08.961362 (+ 2210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34785: BlockingRecv error: recv got EOF from 127.0.98.2:34785 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 28 02:35:09 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:09 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:09.102210 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:09.098244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.3:44443)
0928 02:35:09.098409 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:09.098420 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:09.098475 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:09.098615 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:09.098616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:09.098768 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:09.098816 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:09.098818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:09.099406 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:09.099406 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:09.099934 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:09.099935 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:09.100214 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:09.100215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:09.100252 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:09.100650 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:09.100654 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:09.102153 (+ 1499us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":85,"threads_started":1}
W20260928 02:35:09.102299 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:09.098323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44443 (local address 127.0.98.62:34415)
0928 02:35:09.098382 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:09.098384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:09.098478 (+ 94us) server_negotiation.cc:407] Connection header received
0928 02:35:09.098539 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:09.098539 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:09.098551 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:09.098597 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:09.098867 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:09.099346 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:09.099991 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:09.100164 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:09.100208 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:09.102240 (+ 2032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44443: BlockingRecv error: recv got EOF from 127.0.98.3:44443 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 28 02:35:09 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:09 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:09.822304 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:09.817994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.1:49699)
0928 02:35:09.818112 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:09.818121 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:09.818177 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:09.818303 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:09.818304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:09.818453 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:09.818519 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:09.818521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:09.819109 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:09.819111 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:09.819769 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:09.819772 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:09.820120 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:09.820120 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:09.820161 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:09.820582 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:09.820587 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:09.822243 (+ 1656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
W20260928 02:35:09.822402 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:09.818044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49699 (local address 127.0.98.62:34415)
0928 02:35:09.818149 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:09.818151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:09.818162 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:35:09.818224 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:09.818225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:09.818235 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:09.818287 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:09.818567 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:09.819055 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:09.819836 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:09.820066 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:09.820116 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:09.822324 (+ 2208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49699: BlockingRecv error: recv got EOF from 127.0.98.1:49699 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":59,"threads_started":1}
Sep 28 02:35:09 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:09 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:09.969631 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:09.961759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.2:44451)
0928 02:35:09.961886 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:09.961898 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:09.961973 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:09.962091 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:09.962092 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:09.962245 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:09.962292 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:09.962294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:09.962873 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:09.962873 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:09.963410 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:09.963411 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:09.963737 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:09.963738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:09.963791 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:09.967921 (+ 4130us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:09.967926 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:09.969574 (+ 1648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":76,"threads_started":1}
W20260928 02:35:09.969718 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:09.961817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44451 (local address 127.0.98.62:34415)
0928 02:35:09.961881 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:09.961883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:09.962008 (+ 125us) server_negotiation.cc:407] Connection header received
0928 02:35:09.962028 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:09.962029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:09.962041 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:09.962077 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:09.962340 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:09.962822 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:09.963470 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:09.963650 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:09.963697 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:09.969661 (+ 5964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44451: BlockingRecv error: recv got EOF from 127.0.98.2:44451 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 02:35:10 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:10 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:10.107191 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:10.102692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.3:32897)
0928 02:35:10.102858 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:10.102868 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:10.102935 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:10.103046 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:10.103047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:10.103207 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:10.103258 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:10.103259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:10.103861 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:10.103862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:10.104406 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:10.104407 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:10.104769 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:10.104770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:10.104826 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:10.105232 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:10.105236 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:10.107133 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":72,"threads_started":1}
W20260928 02:35:10.107281 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:10.102775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32897 (local address 127.0.98.62:34415)
0928 02:35:10.102848 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:10.102850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:10.102938 (+ 88us) server_negotiation.cc:407] Connection header received
0928 02:35:10.102976 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:10.102977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:10.102988 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:10.103035 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:10.103306 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:10.103782 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:10.104494 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:10.104684 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:10.104751 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:10.107216 (+ 2465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:32897: BlockingRecv error: recv got EOF from 127.0.98.3:32897 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 02:35:10 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:10 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:10.826761 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:10.822697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.1:42559)
0928 02:35:10.822829 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:10.822839 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:10.822899 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:10.823018 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:10.823019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:10.823169 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:10.823217 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:10.823219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:10.823871 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:10.823872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:10.824396 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:10.824398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:10.824673 (+ 275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:10.824674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:10.824749 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:10.825151 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:10.825155 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:10.826694 (+ 1539us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
W20260928 02:35:10.826838 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:10.822751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42559 (local address 127.0.98.62:34415)
0928 02:35:10.822842 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:10.822843 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:10.822897 (+ 54us) server_negotiation.cc:407] Connection header received
0928 02:35:10.822949 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:10.822949 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:10.822958 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:10.823004 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:10.823300 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:10.823800 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:10.824454 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:10.824629 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:10.824671 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:10.826776 (+ 2105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42559: BlockingRecv error: recv got EOF from 127.0.98.1:42559 (error 108)
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":41,"threads_started":1}
Sep 28 02:35:10 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:10 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:10.974377 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:10.970058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.2:53849)
0928 02:35:10.970165 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:10.970174 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:10.970223 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:10.970349 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:10.970349 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:10.970505 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:10.970551 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:10.970553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:10.971247 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:10.971248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:10.971779 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:10.971781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:10.972138 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:10.972139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:10.972189 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:10.972628 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:10.972633 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:10.974318 (+ 1685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":54,"threads_started":1}
W20260928 02:35:10.974489 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:10.970116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53849 (local address 127.0.98.62:34415)
0928 02:35:10.970196 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:10.970198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:10.970208 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:35:10.970279 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:10.970280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:10.970292 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:10.970338 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:10.970624 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:10.971182 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:10.971852 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:10.972056 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:10.972139 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:10.974430 (+ 2291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53849: BlockingRecv error: recv got EOF from 127.0.98.2:53849 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 02:35:11 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:11 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:11.112102 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:11.107639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.3:49325)
0928 02:35:11.107781 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:11.107793 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:11.107857 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:11.107989 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:11.107990 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:11.108147 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:11.108201 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:11.108203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:11.108798 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:11.108800 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:11.109585 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:11.109588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:11.109893 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:11.109894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:11.109933 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:11.110346 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:11.110349 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:11.112028 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":57,"threads_started":1}
W20260928 02:35:11.112164 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:11.107755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49325 (local address 127.0.98.62:34415)
0928 02:35:11.107816 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:11.107818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:11.107840 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:35:11.107904 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:11.107904 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:11.107916 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:11.107958 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:11.108250 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:11.108728 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:11.109642 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:11.109832 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:11.109887 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:11.112112 (+ 2225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49325: BlockingRecv error: recv got EOF from 127.0.98.3:49325 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 28 02:35:11 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:11 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:11.831720 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:11.827251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.1:35383)
0928 02:35:11.827363 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:11.827376 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:11.827439 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:11.827603 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:11.827604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:11.827755 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:11.827801 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:11.827803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:11.828407 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:11.828408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:11.828931 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:11.828932 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:11.829284 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:11.829286 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:11.829334 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:11.829807 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:11.829811 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:11.831626 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":55,"threads_started":1}
W20260928 02:35:11.831780 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:11.827350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35383 (local address 127.0.98.62:34415)
0928 02:35:11.827490 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:11.827492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:11.827501 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:35:11.827521 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:11.827521 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:11.827531 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:11.827593 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:11.827857 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:11.828350 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:11.829003 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:11.829203 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:11.829289 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:11.831728 (+ 2439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35383: BlockingRecv error: recv got EOF from 127.0.98.1:35383 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":63,"threads_started":1}
Sep 28 02:35:11 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:11 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:11.979182 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:11.974855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.2:42865)
0928 02:35:11.974989 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:11.974999 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:11.975061 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:11.975172 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:11.975173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:11.975329 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:11.975382 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:11.975384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:11.976010 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:11.976011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:11.976544 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:11.976545 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:11.976918 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:11.976919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:11.976954 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:11.977432 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:11.977436 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:11.979121 (+ 1685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":75,"threads_started":1}
W20260928 02:35:11.979278 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:11.974952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42865 (local address 127.0.98.62:34415)
0928 02:35:11.975010 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:11.975012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:11.975053 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:35:11.975102 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:11.975103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:11.975113 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:11.975161 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:11.975431 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:11.975943 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:11.976627 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:11.976852 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:11.976912 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:11.979220 (+ 2308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42865: BlockingRecv error: recv got EOF from 127.0.98.2:42865 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 02:35:12 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:12 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:12.117444 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:12.112551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.3:52587)
0928 02:35:12.112683 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:12.112692 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:12.112758 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:12.112897 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:12.112898 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:12.113044 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:12.113090 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:12.113091 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:12.114032 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:12.114034 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:12.114568 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:12.114570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:12.115045 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:12.115046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:12.115092 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:12.115547 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:12.115551 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:12.117389 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
W20260928 02:35:12.117556 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:12.112627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52587 (local address 127.0.98.62:34415)
0928 02:35:12.112687 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:12.112690 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:12.112753 (+ 63us) server_negotiation.cc:407] Connection header received
0928 02:35:12.112804 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:12.112805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:12.112820 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:12.112885 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:12.113147 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:12.113946 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:12.114670 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:12.114973 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:12.115038 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:12.117490 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52587: BlockingRecv error: recv got EOF from 127.0.98.3:52587 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 28 02:35:12 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:12 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:12.841603 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:12.832129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.1:36183)
0928 02:35:12.832237 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:12.832246 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:12.832293 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:12.832403 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:12.832403 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:12.832549 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:12.832595 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:12.832596 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:12.833204 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:12.833204 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:12.833774 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:12.833776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:12.834065 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:12.834066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:12.834101 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:12.839878 (+ 5777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:12.839882 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:12.841518 (+ 1636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":53,"threads_started":1}
W20260928 02:35:12.841671 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:12.832179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36183 (local address 127.0.98.62:34415)
0928 02:35:12.832283 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:12.832285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:12.832292 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:35:12.832334 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:12.832335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:12.832344 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:12.832398 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:12.832642 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:12.833150 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:12.833834 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:12.834018 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:12.834063 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:12.841613 (+ 7550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36183: BlockingRecv error: recv got EOF from 127.0.98.1:36183 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":51,"threads_started":1}
Sep 28 02:35:12 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:12 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:12.983687 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:12.979558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.2:46817)
0928 02:35:12.979697 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:12.979708 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:12.979769 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:12.979878 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:12.979879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:12.980028 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:12.980109 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:12.980111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:12.980701 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:12.980702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:12.981287 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:12.981288 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:12.981578 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:12.981579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:12.981625 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:12.982017 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:12.982021 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:12.983612 (+ 1591us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":62,"threads_started":1}
W20260928 02:35:12.983747 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:12.979626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46817 (local address 127.0.98.62:34415)
0928 02:35:12.979675 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:12.979677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:12.979752 (+ 75us) server_negotiation.cc:407] Connection header received
0928 02:35:12.979812 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:12.979812 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:12.979822 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:12.979867 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:12.980162 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:12.980639 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:12.981345 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:12.981525 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:12.981575 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:12.983695 (+ 2120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46817: BlockingRecv error: recv got EOF from 127.0.98.2:46817 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 28 02:35:13 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:13 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:13.121999 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:13.117866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.3:32829)
0928 02:35:13.118002 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:13.118011 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:13.118064 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:13.118187 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:13.118188 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:13.118339 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:13.118386 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:13.118388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:13.118998 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:13.118999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:13.119522 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:13.119524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:13.119825 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:13.119827 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:13.119878 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:13.120272 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:13.120276 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:13.121924 (+ 1648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":66,"threads_started":1}
W20260928 02:35:13.122064 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:13.117914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32829 (local address 127.0.98.62:34415)
0928 02:35:13.117962 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:13.117964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:13.118059 (+ 95us) server_negotiation.cc:407] Connection header received
0928 02:35:13.118118 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:13.118119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:13.118130 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:13.118176 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:13.118451 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:13.118938 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:13.119576 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:13.119754 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:13.119797 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:13.122012 (+ 2215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:32829: BlockingRecv error: recv got EOF from 127.0.98.3:32829 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 28 02:35:13 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:13 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:13.846278 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:13.842055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.1:45435)
0928 02:35:13.842209 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:13.842220 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:13.842278 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:13.842397 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:13.842398 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:13.842549 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:13.842596 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:13.842597 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:13.843182 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:13.843183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:13.843705 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:13.843706 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:13.844073 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:13.844074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:13.844108 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:13.844530 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:13.844534 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:13.846220 (+ 1686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":101,"threads_started":1}
W20260928 02:35:13.846386 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:13.842131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45435 (local address 127.0.98.62:34415)
0928 02:35:13.842228 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:13.842230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:13.842268 (+ 38us) server_negotiation.cc:407] Connection header received
0928 02:35:13.842330 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:13.842330 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:13.842340 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:13.842386 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:13.842647 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:13.843129 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:13.843791 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:13.844003 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:13.844073 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:13.846311 (+ 2238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45435: BlockingRecv error: recv got EOF from 127.0.98.1:45435 (error 108)
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":45,"threads_started":1}
Sep 28 02:35:13 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:13 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:13.988363 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:13.984142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.2:57719)
0928 02:35:13.984284 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:13.984296 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:13.984374 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:13.984512 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:13.984514 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:13.984688 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:13.984743 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:13.984746 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:13.985401 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:13.985402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:13.985926 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:13.985927 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:13.986257 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:13.986258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:13.986309 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:13.986728 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:13.986732 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:13.988306 (+ 1574us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
W20260928 02:35:13.988459 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:13.984261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57719 (local address 127.0.98.62:34415)
0928 02:35:13.984315 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:13.984317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:13.984369 (+ 52us) server_negotiation.cc:407] Connection header received
0928 02:35:13.984425 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:13.984426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:13.984437 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:13.984494 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:13.984805 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:13.985341 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:13.985990 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:13.986195 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:13.986249 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:13.988393 (+ 2144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57719: BlockingRecv error: recv got EOF from 127.0.98.2:57719 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 28 02:35:14 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:14 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:14.127154 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:14.122518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.3:38923)
0928 02:35:14.122666 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:14.122676 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:14.122754 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:14.122892 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:14.122892 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:14.123053 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:14.123101 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:14.123102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:14.123850 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:14.123851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:14.124368 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:14.124369 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:14.124701 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:14.124702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:14.124752 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:14.125234 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:14.125238 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:14.127067 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":67,"threads_started":1}
W20260928 02:35:14.127230 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:14.122603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38923 (local address 127.0.98.62:34415)
0928 02:35:14.122685 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:14.122687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:14.122794 (+ 107us) server_negotiation.cc:407] Connection header received
0928 02:35:14.122821 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:14.122822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:14.122836 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:14.122878 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:14.123157 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:14.123789 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:14.124429 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:14.124604 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:14.124667 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:14.127177 (+ 2510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38923: BlockingRecv error: recv got EOF from 127.0.98.3:38923 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 02:35:14 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:14 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:14.851092 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:14.846700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.1:46485)
0928 02:35:14.846822 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:14.846833 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:14.846882 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:14.847068 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:14.847069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:14.847218 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:14.847264 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:14.847266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:14.847890 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:14.847891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:14.848467 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:14.848469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:14.848806 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:14.848807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:14.848888 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:14.849367 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:14.849372 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:14.850973 (+ 1601us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
W20260928 02:35:14.851127 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:14.846778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46485 (local address 127.0.98.62:34415)
0928 02:35:14.846906 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:14.846908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:14.846919 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:35:14.846944 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:14.846945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:14.846957 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:14.847049 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:14.847323 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:14.847839 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:14.848523 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:14.848740 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:14.848797 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:14.851069 (+ 2272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46485: BlockingRecv error: recv got EOF from 127.0.98.1:46485 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
Sep 28 02:35:14 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:14 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:14.992897 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:14.988796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.2:35977)
0928 02:35:14.988927 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:14.988940 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:14.989008 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:14.989144 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:14.989145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:14.989339 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:14.989385 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:14.989387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:14.990014 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:14.990015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:14.990526 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:14.990527 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:14.990842 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:14.990843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:14.990892 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:14.991290 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:14.991294 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:14.992822 (+ 1528us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
W20260928 02:35:14.992965 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:14.988881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35977 (local address 127.0.98.62:34415)
0928 02:35:14.988943 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:14.988945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:14.988989 (+ 44us) server_negotiation.cc:407] Connection header received
0928 02:35:14.989059 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:14.989060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:14.989071 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:14.989132 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:14.989462 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:14.989945 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:14.990598 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:14.990787 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:14.990840 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:14.992913 (+ 2073us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35977: BlockingRecv error: recv got EOF from 127.0.98.2:35977 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 02:35:15 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:15 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:15.131669 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:15.127582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.3:45023)
0928 02:35:15.127697 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:15.127707 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:15.127767 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:15.127895 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:15.127896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:15.128056 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:15.128117 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:15.128119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:15.128745 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:15.128746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:15.129299 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:15.129302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:15.129623 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:15.129624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:15.129681 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:15.130103 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:15.130107 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:15.131582 (+ 1475us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":62,"threads_started":1}
W20260928 02:35:15.131757 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:15.127683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45023 (local address 127.0.98.62:34415)
0928 02:35:15.127740 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:15.127741 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:15.127749 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:35:15.127822 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:15.127823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:15.127835 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:15.127884 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:15.128187 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:15.128678 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:15.129363 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:15.129561 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:15.129620 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:15.131684 (+ 2064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45023: BlockingRecv error: recv got EOF from 127.0.98.3:45023 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 28 02:35:15 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:15 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:15.855880 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:15.851552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.1:59459)
0928 02:35:15.851673 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:15.851684 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:15.851744 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:15.851858 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:15.851859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:15.852048 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:15.852100 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:15.852102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:15.852805 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:15.852806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:15.853345 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:15.853347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:15.853682 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:15.853684 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:15.853737 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:15.854164 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:15.854168 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:15.855812 (+ 1644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":66,"threads_started":1}
W20260928 02:35:15.855975 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:15.851599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59459 (local address 127.0.98.62:34415)
0928 02:35:15.851691 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:15.851692 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:15.851739 (+ 47us) server_negotiation.cc:407] Connection header received
0928 02:35:15.851794 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:15.851794 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:15.851804 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:15.851848 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:15.852170 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:15.852727 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:15.853413 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:15.853623 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:15.853678 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:15.855910 (+ 2232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59459: BlockingRecv error: recv got EOF from 127.0.98.1:59459 (error 108)
Metrics: {"server-negotiator.queue_time_us":72,"thread_start_us":42,"threads_started":1}
Sep 28 02:35:15 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:35:15 dist-test-slave-fz47 krb5kdc[8528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:35:15.997440 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:15.993442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34415 (local address 127.0.98.2:33767)
0928 02:35:15.993572 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:15.993581 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:15.993637 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:15.993760 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:15.993761 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:15.993917 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:15.993968 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:15.993970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:15.994577 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:15.994578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:15.995107 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:15.995108 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:15.995397 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:15.995397 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:15.995436 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:15.995842 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:15.995846 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:15.997373 (+ 1527us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34415: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
W20260928 02:35:15.997536 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:15.993491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33767 (local address 127.0.98.62:34415)
0928 02:35:15.993579 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:15.993581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:15.993623 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:35:15.993685 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:15.993685 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:15.993697 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:15.993749 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:15.994021 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:15.994496 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:15.995166 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:15.995346 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:15.995393 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:15.997454 (+ 2061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33767: BlockingRecv error: recv got EOF from 127.0.98.2:33767 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:35:16.064348 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 9111
I20260928 02:35:16.068149 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 9263
I20260928 02:35:16.071882 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 9397
I20260928 02:35:16.075093 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 9041
2026-09-28T02:35:16Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17320 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T02:35:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:35:16Z Disabled control of system clock
I20260928 02:35:16.099494 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36517
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 02:35:16.175124 9593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:35:16.175314 9593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:35:16.175339 9593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:35:16.176478 9593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:35:16.176514 9593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:35:16.176530 9593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:35:16.176545 9593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:35:16.176559 9593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:35:16.177865 9593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36517
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:35:16.178092 9593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:35:16.178344 9593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:35:16.183185 9601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:35:16.183205 9599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:35:16.183300 9593 server_base.cc:1034] running on GCE node
W20260928 02:35:16.183251 9598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:35:16.183629 9593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:35:16.183879 9593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:35:16.185014 9593 hybrid_clock.cc:648] HybridClock initialized: now 1790562916184998 us; error 37 us; skew 500 ppm
I20260928 02:35:16.186126 9593 webserver.cc:460] Webserver started at http://127.0.98.62:44755/ using document root <none> and password file <none>
I20260928 02:35:16.186328 9593 fs_manager.cc:359] Metadata directory not provided
I20260928 02:35:16.186374 9593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:35:16.186493 9593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:35:16.187477 9593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "79219ef8eb894e93973d38716df8206a"
format_stamp: "Formatted at 2026-09-28 02:35:16 on dist-test-slave-fz47"
server_key: "db9a5304207fcd3249923716f9fb1ea9"
server_key_iv: "89c56db55b0aa2bdabfd0593b802188a"
server_key_version: "encryptionkey@0"
I20260928 02:35:16.187796 9593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "79219ef8eb894e93973d38716df8206a"
format_stamp: "Formatted at 2026-09-28 02:35:16 on dist-test-slave-fz47"
server_key: "db9a5304207fcd3249923716f9fb1ea9"
server_key_iv: "89c56db55b0aa2bdabfd0593b802188a"
server_key_version: "encryptionkey@0"
I20260928 02:35:16.188913 9593 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:35:16.189595 9607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:16.189738 9593 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:16.189815 9593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "79219ef8eb894e93973d38716df8206a"
format_stamp: "Formatted at 2026-09-28 02:35:16 on dist-test-slave-fz47"
server_key: "db9a5304207fcd3249923716f9fb1ea9"
server_key_iv: "89c56db55b0aa2bdabfd0593b802188a"
server_key_version: "encryptionkey@0"
I20260928 02:35:16.189877 9593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:35:16.207931 9593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:35:16.208190 9593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:35:16.208307 9593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:35:16.212105 9593 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36517
I20260928 02:35:16.212136 9659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36517 every 8 connection(s)
I20260928 02:35:16.212424 9593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:35:16.212934 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 9593
I20260928 02:35:16.213011 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:35:16.213083 9660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:35:16.213197 392 external_mini_cluster.cc:1428] Setting key f1b0792e0a55e71863b81d3cd3d13483
I20260928 02:35:16.216670 9660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79219ef8eb894e93973d38716df8206a: Bootstrap starting.
I20260928 02:35:16.217285 9660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79219ef8eb894e93973d38716df8206a: Neither blocks nor log segments found. Creating new log.
I20260928 02:35:16.217602 9660 log.cc:824] T 00000000000000000000000000000000 P 79219ef8eb894e93973d38716df8206a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:35:16.218321 9660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79219ef8eb894e93973d38716df8206a: No bootstrap required, opened a new log
I20260928 02:35:16.219061 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:16.213710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39656 (local address 127.0.98.62:36517)
0928 02:35:16.213933 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:16.213942 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:16.213981 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:35:16.214577 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:16.214587 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:16.214672 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:16.214819 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:16.214990 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:16.216263 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:16.217072 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:16.217681 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:16.217797 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:16.218398 (+ 601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:35:16.218403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:35:16.218403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:35:16.218606 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:35:16.218645 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:35:16.218659 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:35:16.218723 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:35:16.218782 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 02:35:16.218859 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":95,"threads_started":1}
I20260928 02:35:16.220309 9660 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79219ef8eb894e93973d38716df8206a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79219ef8eb894e93973d38716df8206a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36517 } }
I20260928 02:35:16.220435 9660 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79219ef8eb894e93973d38716df8206a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:35:16.220458 9660 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79219ef8eb894e93973d38716df8206a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79219ef8eb894e93973d38716df8206a, State: Initialized, Role: FOLLOWER
I20260928 02:35:16.220546 9660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79219ef8eb894e93973d38716df8206a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79219ef8eb894e93973d38716df8206a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36517 } }
I20260928 02:35:16.220620 9660 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79219ef8eb894e93973d38716df8206a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:35:16.220640 9660 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79219ef8eb894e93973d38716df8206a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:35:16.220671 9660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79219ef8eb894e93973d38716df8206a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:35:16.221138 9660 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79219ef8eb894e93973d38716df8206a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79219ef8eb894e93973d38716df8206a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36517 } }
I20260928 02:35:16.221218 9660 leader_election.cc:302] T 00000000000000000000000000000000 P 79219ef8eb894e93973d38716df8206a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79219ef8eb894e93973d38716df8206a; no voters:
I20260928 02:35:16.221365 9660 leader_election.cc:288] T 00000000000000000000000000000000 P 79219ef8eb894e93973d38716df8206a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:35:16.221422 9665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79219ef8eb894e93973d38716df8206a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:35:16.221580 9665 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79219ef8eb894e93973d38716df8206a [term 1 LEADER]: Becoming Leader. State: Replica: 79219ef8eb894e93973d38716df8206a, State: Running, Role: LEADER
I20260928 02:35:16.221599 9660 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79219ef8eb894e93973d38716df8206a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:35:16.221722 9665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79219ef8eb894e93973d38716df8206a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79219ef8eb894e93973d38716df8206a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36517 } }
I20260928 02:35:16.222105 9666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79219ef8eb894e93973d38716df8206a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79219ef8eb894e93973d38716df8206a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79219ef8eb894e93973d38716df8206a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36517 } } }
I20260928 02:35:16.222118 9667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79219ef8eb894e93973d38716df8206a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79219ef8eb894e93973d38716df8206a. Latest consensus state: current_term: 1 leader_uuid: "79219ef8eb894e93973d38716df8206a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79219ef8eb894e93973d38716df8206a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36517 } } }
I20260928 02:35:16.222194 9666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79219ef8eb894e93973d38716df8206a [sys.catalog]: This master's current role is: LEADER
I20260928 02:35:16.222203 9667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79219ef8eb894e93973d38716df8206a [sys.catalog]: This master's current role is: LEADER
I20260928 02:35:16.222728 9673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:35:16.223114 9673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:35:16.224540 9673 catalog_manager.cc:1347] Generated new cluster ID: 4cc8aa7f158c439bb7dc6f2f3b3f3e1f
I20260928 02:35:16.224581 9673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:35:16.237725 9673 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:35:16.238197 9673 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:35:16.253021 9673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79219ef8eb894e93973d38716df8206a: Generated new TSK 0
I20260928 02:35:16.253284 9673 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:35:16.264972 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36517
--builtin_ntp_servers=127.0.98.20:36389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 02:35:16.341795 9684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:35:16.341984 9684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:35:16.342011 9684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:35:16.343110 9684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:35:16.343147 9684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:35:16.343179 9684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:35:16.344453 9684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:35:16.344664 9684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:35:16.344921 9684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:35:16.349491 9690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:35:16.349525 9692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:35:16.349560 9684 server_base.cc:1034] running on GCE node
W20260928 02:35:16.349529 9689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:35:16.349829 9684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:35:16.350039 9684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:35:16.351166 9684 hybrid_clock.cc:648] HybridClock initialized: now 1790562916351150 us; error 35 us; skew 500 ppm
I20260928 02:35:16.352237 9684 webserver.cc:460] Webserver started at http://127.0.98.1:45845/ using document root <none> and password file <none>
I20260928 02:35:16.352453 9684 fs_manager.cc:359] Metadata directory not provided
I20260928 02:35:16.352505 9684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:35:16.352628 9684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:35:16.353705 9684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/data/instance:
uuid: "8bf89697b8114cf591bcc96baa08ab15"
format_stamp: "Formatted at 2026-09-28 02:35:16 on dist-test-slave-fz47"
server_key: "45edb49193ab5c39af8af902d4fb0267"
server_key_iv: "8c99d10d76daae45651f00dc43268b00"
server_key_version: "encryptionkey@0"
I20260928 02:35:16.354012 9684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8bf89697b8114cf591bcc96baa08ab15"
format_stamp: "Formatted at 2026-09-28 02:35:16 on dist-test-slave-fz47"
server_key: "45edb49193ab5c39af8af902d4fb0267"
server_key_iv: "8c99d10d76daae45651f00dc43268b00"
server_key_version: "encryptionkey@0"
I20260928 02:35:16.355142 9684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:35:16.355789 9698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:16.355922 9684 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:35:16.355990 9684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "8bf89697b8114cf591bcc96baa08ab15"
format_stamp: "Formatted at 2026-09-28 02:35:16 on dist-test-slave-fz47"
server_key: "45edb49193ab5c39af8af902d4fb0267"
server_key_iv: "8c99d10d76daae45651f00dc43268b00"
server_key_version: "encryptionkey@0"
I20260928 02:35:16.356055 9684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:35:16.381904 9684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:35:16.382184 9684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:35:16.382308 9684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:35:16.382529 9684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:35:16.382826 9684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:35:16.382861 9684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:16.382889 9684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:35:16.382912 9684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:16.387974 9684 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44419
I20260928 02:35:16.388042 9811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44419 every 8 connection(s)
I20260928 02:35:16.388303 9684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
I20260928 02:35:16.389624 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 9684
I20260928 02:35:16.389703 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/wal/instance
I20260928 02:35:16.389870 392 external_mini_cluster.cc:1428] Setting key 6fc79ebbb981761385a0d328fed1284d
I20260928 02:35:16.390717 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36517
--builtin_ntp_servers=127.0.98.20:36389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 02:35:16.393419 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:16.388970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55041 (local address 127.0.98.62:36517)
0928 02:35:16.389034 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:16.389036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:16.389242 (+ 206us) server_negotiation.cc:407] Connection header received
0928 02:35:16.389804 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:16.389806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:16.389822 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:16.389876 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:16.390585 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:16.391072 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:16.392033 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:16.392269 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:16.392322 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:16.393106 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:35:16.393108 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:35:16.393108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:35:16.393112 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:35:16.393126 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:35:16.393129 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:35:16.393175 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 02:35:16.393321 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 02:35:16.393363 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:35:16.393648 9814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:16.388949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36517 (local address 127.0.98.1:55041)
0928 02:35:16.389133 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:16.389163 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:16.389686 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:16.389946 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:16.389950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:16.389961 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:16.390514 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:16.390520 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:16.391148 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:16.391149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:16.391953 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:16.391956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:16.392337 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:16.392338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:16.392472 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:16.392974 (+ 502us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:35:16.392975 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:16.393005 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:35:16.393029 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:35:16.393200 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:35:16.393206 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:35:16.393209 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:35:16.393319 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 02:35:16.393423 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
I20260928 02:35:16.394141 9812 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36517
I20260928 02:35:16.394234 9812 heartbeater.cc:464] Registering TS with master...
I20260928 02:35:16.394415 9812 heartbeater.cc:511] Master 127.0.98.62:36517 requested a full tablet report, sending...
I20260928 02:35:16.394881 9624 ts_manager.cc:194] Registered new tserver with Master: 8bf89697b8114cf591bcc96baa08ab15 (127.0.98.1:44419)
I20260928 02:35:16.395464 9624 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:55041
W20260928 02:35:16.467859 9815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:35:16.468056 9815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:35:16.468076 9815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:35:16.469202 9815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:35:16.469244 9815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:35:16.469282 9815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:35:16.470579 9815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:35:16.470794 9815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:35:16.471060 9815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:35:16.476981 9821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:35:16.477052 9823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:35:16.476989 9820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:35:16.477151 9815 server_base.cc:1034] running on GCE node
I20260928 02:35:16.477372 9815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:35:16.477613 9815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:35:16.478744 9815 hybrid_clock.cc:648] HybridClock initialized: now 1790562916478734 us; error 26 us; skew 500 ppm
I20260928 02:35:16.479907 9815 webserver.cc:460] Webserver started at http://127.0.98.2:38533/ using document root <none> and password file <none>
I20260928 02:35:16.480118 9815 fs_manager.cc:359] Metadata directory not provided
I20260928 02:35:16.480160 9815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:35:16.480273 9815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:35:16.481348 9815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/data/instance:
uuid: "3d34db5a69cf453f9925173e89fa6e85"
format_stamp: "Formatted at 2026-09-28 02:35:16 on dist-test-slave-fz47"
server_key: "2f0d2083e27c1de0c686b54b496e9d16"
server_key_iv: "53ae7ff05482cb46f2d7f83daaef0b05"
server_key_version: "encryptionkey@0"
I20260928 02:35:16.481678 9815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3d34db5a69cf453f9925173e89fa6e85"
format_stamp: "Formatted at 2026-09-28 02:35:16 on dist-test-slave-fz47"
server_key: "2f0d2083e27c1de0c686b54b496e9d16"
server_key_iv: "53ae7ff05482cb46f2d7f83daaef0b05"
server_key_version: "encryptionkey@0"
I20260928 02:35:16.482771 9815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:35:16.483352 9829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:16.483469 9815 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:16.483539 9815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "3d34db5a69cf453f9925173e89fa6e85"
format_stamp: "Formatted at 2026-09-28 02:35:16 on dist-test-slave-fz47"
server_key: "2f0d2083e27c1de0c686b54b496e9d16"
server_key_iv: "53ae7ff05482cb46f2d7f83daaef0b05"
server_key_version: "encryptionkey@0"
I20260928 02:35:16.483597 9815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:35:16.495437 9815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:35:16.495684 9815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:35:16.495797 9815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:35:16.495966 9815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:35:16.496259 9815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:35:16.496290 9815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:16.496310 9815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:35:16.496332 9815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:16.501417 9815 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34929
I20260928 02:35:16.501458 9942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34929 every 8 connection(s)
I20260928 02:35:16.501718 9815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
I20260928 02:35:16.504940 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 9815
I20260928 02:35:16.505015 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/wal/instance
I20260928 02:35:16.505182 392 external_mini_cluster.cc:1428] Setting key 05270aa9c85637caecac9f616344b73c
I20260928 02:35:16.506019 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36517
--builtin_ntp_servers=127.0.98.20:36389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 02:35:16.506443 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:16.502321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52115 (local address 127.0.98.62:36517)
0928 02:35:16.502368 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:16.502370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:16.502583 (+ 213us) server_negotiation.cc:407] Connection header received
0928 02:35:16.503055 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:16.503056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:16.503068 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:16.503117 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:16.503722 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:16.504200 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:16.505139 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:16.505398 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:16.505451 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:16.506193 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:35:16.506195 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:35:16.506195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:35:16.506199 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:35:16.506214 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:35:16.506218 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:35:16.506240 (+ 22us) server_negotiation.cc:1035] Waiting for connection context
0928 02:35:16.506344 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 02:35:16.506367 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 02:35:16.506616 9945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:16.502300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36517 (local address 127.0.98.2:52115)
0928 02:35:16.502475 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:16.502511 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:16.502963 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:16.503181 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:16.503185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:16.503194 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:16.503645 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:16.503648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:16.504279 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:16.504280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:16.505063 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:16.505065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:16.505476 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:16.505477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:16.505599 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:16.506049 (+ 450us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:35:16.506050 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:16.506077 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:35:16.506098 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:35:16.506275 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:35:16.506279 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:35:16.506281 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:35:16.506331 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 02:35:16.506425 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260928 02:35:16.507025 9943 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36517
I20260928 02:35:16.507109 9943 heartbeater.cc:464] Registering TS with master...
I20260928 02:35:16.507319 9943 heartbeater.cc:511] Master 127.0.98.62:36517 requested a full tablet report, sending...
I20260928 02:35:16.507628 9624 ts_manager.cc:194] Registered new tserver with Master: 3d34db5a69cf453f9925173e89fa6e85 (127.0.98.2:34929)
I20260928 02:35:16.507972 9624 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:52115
W20260928 02:35:16.582581 9946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:35:16.582759 9946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:35:16.582778 9946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:35:16.583868 9946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:35:16.583902 9946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:35:16.583925 9946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:35:16.585178 9946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:35:16.585408 9946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:35:16.585675 9946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:35:16.590358 9952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:35:16.590409 9951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:35:16.590369 9954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:35:16.590536 9946 server_base.cc:1034] running on GCE node
I20260928 02:35:16.590790 9946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:35:16.591020 9946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:35:16.592159 9946 hybrid_clock.cc:648] HybridClock initialized: now 1790562916592141 us; error 26 us; skew 500 ppm
I20260928 02:35:16.593398 9946 webserver.cc:460] Webserver started at http://127.0.98.3:41983/ using document root <none> and password file <none>
I20260928 02:35:16.593626 9946 fs_manager.cc:359] Metadata directory not provided
I20260928 02:35:16.593674 9946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:35:16.593794 9946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:35:16.594790 9946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/data/instance:
uuid: "a56d6f6c9ab34885a1d8b1ee0abe53ba"
format_stamp: "Formatted at 2026-09-28 02:35:16 on dist-test-slave-fz47"
server_key: "e46874004aa2f1782fc933a007bf69c6"
server_key_iv: "51c1f530158404420c55919d75241eab"
server_key_version: "encryptionkey@0"
I20260928 02:35:16.595110 9946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a56d6f6c9ab34885a1d8b1ee0abe53ba"
format_stamp: "Formatted at 2026-09-28 02:35:16 on dist-test-slave-fz47"
server_key: "e46874004aa2f1782fc933a007bf69c6"
server_key_iv: "51c1f530158404420c55919d75241eab"
server_key_version: "encryptionkey@0"
I20260928 02:35:16.596266 9946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:35:16.596868 9960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:16.597012 9946 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:35:16.597088 9946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "a56d6f6c9ab34885a1d8b1ee0abe53ba"
format_stamp: "Formatted at 2026-09-28 02:35:16 on dist-test-slave-fz47"
server_key: "e46874004aa2f1782fc933a007bf69c6"
server_key_iv: "51c1f530158404420c55919d75241eab"
server_key_version: "encryptionkey@0"
I20260928 02:35:16.597150 9946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:35:16.616223 9946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:35:16.616497 9946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:35:16.616618 9946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:35:16.616806 9946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:35:16.617108 9946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:35:16.617141 9946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:16.617169 9946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:35:16.617193 9946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:16.622499 9946 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44463
I20260928 02:35:16.622550 10073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44463 every 8 connection(s)
I20260928 02:35:16.622844 9946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
I20260928 02:35:16.627468 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:16.623540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44077 (local address 127.0.98.62:36517)
0928 02:35:16.623592 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:16.623595 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:16.623832 (+ 237us) server_negotiation.cc:407] Connection header received
0928 02:35:16.624286 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:16.624286 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:16.624296 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:16.624345 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:16.624937 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:16.625469 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:16.626369 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:16.626559 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:16.626602 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:16.627259 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:35:16.627260 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:35:16.627261 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:35:16.627264 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:35:16.627277 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:35:16.627281 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:35:16.627317 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 02:35:16.627375 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 02:35:16.627394 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:35:16.627625 10076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:16.623535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36517 (local address 127.0.98.3:44077)
0928 02:35:16.623719 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:16.623751 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:16.624214 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:16.624399 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:16.624402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:16.624411 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:16.624872 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:16.624876 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:16.625536 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:16.625536 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:16.626310 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:16.626312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:16.626612 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:16.626613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:16.626711 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:16.627146 (+ 435us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:35:16.627146 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:16.627173 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:35:16.627193 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:35:16.627322 (+ 129us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:35:16.627324 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:35:16.627326 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:35:16.627370 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 02:35:16.627441 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
I20260928 02:35:16.628062 10074 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36517
I20260928 02:35:16.628145 10074 heartbeater.cc:464] Registering TS with master...
I20260928 02:35:16.628311 10074 heartbeater.cc:511] Master 127.0.98.62:36517 requested a full tablet report, sending...
I20260928 02:35:16.628645 9624 ts_manager.cc:194] Registered new tserver with Master: a56d6f6c9ab34885a1d8b1ee0abe53ba (127.0.98.3:44463)
I20260928 02:35:16.628991 9624 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:44077
I20260928 02:35:16.630403 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 9946
I20260928 02:35:16.630483 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/wal/instance
I20260928 02:35:16.630630 392 external_mini_cluster.cc:1428] Setting key ce425e2a6088db5205e3198a2d9543ec
I20260928 02:35:16.631358 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 02:35:16.632910 9663 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:16.632517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39668 (local address 127.0.98.62:36517)
0928 02:35:16.632561 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:16.632563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:16.632681 (+ 118us) server_negotiation.cc:407] Connection header received
0928 02:35:16.632704 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:16.632704 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:16.632713 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:16.632751 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:16.632854 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39668: BlockingRecv error: recv got EOF from 127.0.0.1:39668 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 02:35:16.632999 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 9684
I20260928 02:35:16.637349 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 9815
I20260928 02:35:16.641129 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 9946
I20260928 02:35:16.644456 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 9593
2026-09-28T02:35:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (568 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T02:35:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:35:16Z Disabled control of system clock
I20260928 02:35:16.668129 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42197
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 02:35:16.745189 10085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:35:16.745400 10085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:35:16.745420 10085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:35:16.746529 10085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:35:16.746567 10085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:35:16.746578 10085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:35:16.746589 10085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:35:16.746600 10085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:35:16.747897 10085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42197
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:35:16.748127 10085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:35:16.748404 10085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:35:16.752997 10093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:35:16.753139 10085 server_base.cc:1034] running on GCE node
W20260928 02:35:16.753005 10090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:35:16.753892 10091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:35:16.754169 10085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:35:16.754447 10085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:35:16.755586 10085 hybrid_clock.cc:648] HybridClock initialized: now 1790562916755573 us; error 29 us; skew 500 ppm
I20260928 02:35:16.756733 10085 webserver.cc:460] Webserver started at http://127.0.98.62:39857/ using document root <none> and password file <none>
I20260928 02:35:16.756924 10085 fs_manager.cc:359] Metadata directory not provided
I20260928 02:35:16.756963 10085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:35:16.757047 10085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:35:16.758042 10085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "3eee39efcedd43e49d3f48e1d918b2c4"
format_stamp: "Formatted at 2026-09-28 02:35:16 on dist-test-slave-fz47"
server_key: "4c1421ec358ca46da37a36ff64645e25"
server_key_iv: "d23a729456a73661b50389f20e200028"
server_key_version: "encryptionkey@0"
I20260928 02:35:16.758312 10085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "3eee39efcedd43e49d3f48e1d918b2c4"
format_stamp: "Formatted at 2026-09-28 02:35:16 on dist-test-slave-fz47"
server_key: "4c1421ec358ca46da37a36ff64645e25"
server_key_iv: "d23a729456a73661b50389f20e200028"
server_key_version: "encryptionkey@0"
I20260928 02:35:16.759399 10085 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 02:35:16.759992 10099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:16.760133 10085 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:35:16.760196 10085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "3eee39efcedd43e49d3f48e1d918b2c4"
format_stamp: "Formatted at 2026-09-28 02:35:16 on dist-test-slave-fz47"
server_key: "4c1421ec358ca46da37a36ff64645e25"
server_key_iv: "d23a729456a73661b50389f20e200028"
server_key_version: "encryptionkey@0"
I20260928 02:35:16.760260 10085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:35:16.763324 10085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:35:16.763557 10085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:35:16.763654 10085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:35:16.767055 10085 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42197
I20260928 02:35:16.767082 10151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42197 every 8 connection(s)
I20260928 02:35:16.767339 10085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:35:16.767871 10152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:35:16.770370 10152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3eee39efcedd43e49d3f48e1d918b2c4: Bootstrap starting.
I20260928 02:35:16.770859 10152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3eee39efcedd43e49d3f48e1d918b2c4: Neither blocks nor log segments found. Creating new log.
I20260928 02:35:16.771073 10152 log.cc:824] T 00000000000000000000000000000000 P 3eee39efcedd43e49d3f48e1d918b2c4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:35:16.771553 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 10085
I20260928 02:35:16.771656 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:35:16.771695 10152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3eee39efcedd43e49d3f48e1d918b2c4: No bootstrap required, opened a new log
I20260928 02:35:16.771828 392 external_mini_cluster.cc:1428] Setting key 663e0bc61fa68e4789501cd54e4e740f
I20260928 02:35:16.773762 10152 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3eee39efcedd43e49d3f48e1d918b2c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eee39efcedd43e49d3f48e1d918b2c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42197 } }
I20260928 02:35:16.773917 10152 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3eee39efcedd43e49d3f48e1d918b2c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:35:16.774060 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:16.772282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40902 (local address 127.0.98.62:42197)
0928 02:35:16.772448 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:16.772455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:16.772481 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:35:16.772874 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:16.772880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:16.772913 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:16.773022 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:16.773624 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:35:16.773627 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:35:16.773627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:35:16.773630 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:35:16.773665 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:35:16.773675 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:35:16.773724 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:35:16.773795 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 02:35:16.773842 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
I20260928 02:35:16.773970 10152 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3eee39efcedd43e49d3f48e1d918b2c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eee39efcedd43e49d3f48e1d918b2c4, State: Initialized, Role: FOLLOWER
I20260928 02:35:16.774245 10152 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3eee39efcedd43e49d3f48e1d918b2c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eee39efcedd43e49d3f48e1d918b2c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42197 } }
I20260928 02:35:16.774336 10152 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3eee39efcedd43e49d3f48e1d918b2c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:35:16.774364 10152 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3eee39efcedd43e49d3f48e1d918b2c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:35:16.774406 10152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3eee39efcedd43e49d3f48e1d918b2c4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:35:16.775156 10152 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3eee39efcedd43e49d3f48e1d918b2c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eee39efcedd43e49d3f48e1d918b2c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42197 } }
I20260928 02:35:16.775234 10152 leader_election.cc:302] T 00000000000000000000000000000000 P 3eee39efcedd43e49d3f48e1d918b2c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3eee39efcedd43e49d3f48e1d918b2c4; no voters:
I20260928 02:35:16.775360 10152 leader_election.cc:288] T 00000000000000000000000000000000 P 3eee39efcedd43e49d3f48e1d918b2c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:35:16.775384 10157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3eee39efcedd43e49d3f48e1d918b2c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:35:16.775509 10157 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3eee39efcedd43e49d3f48e1d918b2c4 [term 1 LEADER]: Becoming Leader. State: Replica: 3eee39efcedd43e49d3f48e1d918b2c4, State: Running, Role: LEADER
I20260928 02:35:16.775600 10152 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3eee39efcedd43e49d3f48e1d918b2c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:35:16.775599 10157 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3eee39efcedd43e49d3f48e1d918b2c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eee39efcedd43e49d3f48e1d918b2c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42197 } }
I20260928 02:35:16.775990 10159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eee39efcedd43e49d3f48e1d918b2c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3eee39efcedd43e49d3f48e1d918b2c4. Latest consensus state: current_term: 1 leader_uuid: "3eee39efcedd43e49d3f48e1d918b2c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eee39efcedd43e49d3f48e1d918b2c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42197 } } }
I20260928 02:35:16.776069 10159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eee39efcedd43e49d3f48e1d918b2c4 [sys.catalog]: This master's current role is: LEADER
I20260928 02:35:16.776284 10165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:35:16.776229 10158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eee39efcedd43e49d3f48e1d918b2c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3eee39efcedd43e49d3f48e1d918b2c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eee39efcedd43e49d3f48e1d918b2c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42197 } } }
I20260928 02:35:16.776445 10158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eee39efcedd43e49d3f48e1d918b2c4 [sys.catalog]: This master's current role is: LEADER
I20260928 02:35:16.776767 10165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:35:16.778101 10165 catalog_manager.cc:1347] Generated new cluster ID: f2a077ddcdf84e32b74f4509a1ef2824
I20260928 02:35:16.778153 10165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:35:16.790908 10165 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:35:16.791054 10165 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:35:16.795382 10165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3eee39efcedd43e49d3f48e1d918b2c4: Generated new TSK 0
I20260928 02:35:16.795536 10165 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:35:16.798568 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42197
--builtin_ntp_servers=127.0.98.20:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 02:35:16.875753 10176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:35:16.875968 10176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:35:16.875989 10176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:35:16.877177 10176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:35:16.877213 10176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:35:16.877238 10176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:35:16.878572 10176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:35:16.878777 10176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:35:16.879081 10176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:35:16.883913 10181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:35:16.884012 10176 server_base.cc:1034] running on GCE node
W20260928 02:35:16.883880 10182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:35:16.883880 10184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:35:16.884318 10176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:35:16.884616 10176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:35:16.885756 10176 hybrid_clock.cc:648] HybridClock initialized: now 1790562916885744 us; error 26 us; skew 500 ppm
I20260928 02:35:16.887033 10176 webserver.cc:460] Webserver started at http://127.0.98.1:43681/ using document root <none> and password file <none>
I20260928 02:35:16.887276 10176 fs_manager.cc:359] Metadata directory not provided
I20260928 02:35:16.887342 10176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:35:16.887471 10176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:35:16.888852 10176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/data/instance:
uuid: "75bbdbd9c547403097976dd2802a4d0c"
format_stamp: "Formatted at 2026-09-28 02:35:16 on dist-test-slave-fz47"
server_key: "15f89c9c5b8ba478dc3cf4a7d0219cf3"
server_key_iv: "1561402ffdae78b4bfb7aba217f01a40"
server_key_version: "encryptionkey@0"
I20260928 02:35:16.889240 10176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/wal/instance:
uuid: "75bbdbd9c547403097976dd2802a4d0c"
format_stamp: "Formatted at 2026-09-28 02:35:16 on dist-test-slave-fz47"
server_key: "15f89c9c5b8ba478dc3cf4a7d0219cf3"
server_key_iv: "1561402ffdae78b4bfb7aba217f01a40"
server_key_version: "encryptionkey@0"
I20260928 02:35:16.890852 10176 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:35:16.891672 10190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:16.891829 10176 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:35:16.891918 10176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "75bbdbd9c547403097976dd2802a4d0c"
format_stamp: "Formatted at 2026-09-28 02:35:16 on dist-test-slave-fz47"
server_key: "15f89c9c5b8ba478dc3cf4a7d0219cf3"
server_key_iv: "1561402ffdae78b4bfb7aba217f01a40"
server_key_version: "encryptionkey@0"
I20260928 02:35:16.891987 10176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:35:16.896973 10176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:35:16.897223 10176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:35:16.897367 10176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:35:16.897559 10176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:35:16.897840 10176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:35:16.897876 10176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:16.897905 10176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:35:16.897928 10176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:16.904426 10176 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32903
I20260928 02:35:16.904457 10303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32903 every 8 connection(s)
I20260928 02:35:16.904788 10176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
I20260928 02:35:16.907467 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:16.905546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44187 (local address 127.0.98.62:42197)
0928 02:35:16.905608 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:16.905611 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:16.905820 (+ 209us) server_negotiation.cc:407] Connection header received
0928 02:35:16.906237 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:16.906238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:16.906245 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:16.906300 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:16.907296 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:35:16.907297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:35:16.907297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:35:16.907300 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:35:16.907310 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:35:16.907313 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:35:16.907349 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 02:35:16.907393 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 02:35:16.907403 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:35:16.907624 10306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:16.905521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42197 (local address 127.0.98.1:44187)
0928 02:35:16.905695 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:16.905727 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:16.906165 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:16.906353 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:16.906356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:16.906364 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:16.907181 (+ 817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:35:16.907182 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:16.907208 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:35:16.907236 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:35:16.907348 (+ 112us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:35:16.907350 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:35:16.907352 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:35:16.907401 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 02:35:16.907447 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
I20260928 02:35:16.908031 10304 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42197
I20260928 02:35:16.908113 10304 heartbeater.cc:464] Registering TS with master...
I20260928 02:35:16.908245 10304 heartbeater.cc:511] Master 127.0.98.62:42197 requested a full tablet report, sending...
I20260928 02:35:16.908690 10116 ts_manager.cc:194] Registered new tserver with Master: 75bbdbd9c547403097976dd2802a4d0c (127.0.98.1:32903)
I20260928 02:35:16.912578 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 10176
I20260928 02:35:16.912654 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-0/wal/instance
I20260928 02:35:16.912819 392 external_mini_cluster.cc:1428] Setting key 3fd2b6b671a18e52f616de8dfa0bb6d9
I20260928 02:35:16.913594 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42197
--builtin_ntp_servers=127.0.98.20:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 02:35:16.991540 10307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:35:16.991725 10307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:35:16.991745 10307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:35:16.992861 10307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:35:16.992898 10307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:35:16.992923 10307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:35:16.994251 10307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:35:16.994454 10307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:35:16.994711 10307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:35:16.999460 10313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:35:16.999473 10315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:35:16.999528 10307 server_base.cc:1034] running on GCE node
W20260928 02:35:16.999650 10312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:35:16.999867 10307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:35:17.000090 10307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:35:17.001238 10307 hybrid_clock.cc:648] HybridClock initialized: now 1790562917001223 us; error 34 us; skew 500 ppm
I20260928 02:35:17.002466 10307 webserver.cc:460] Webserver started at http://127.0.98.2:44575/ using document root <none> and password file <none>
I20260928 02:35:17.002667 10307 fs_manager.cc:359] Metadata directory not provided
I20260928 02:35:17.002713 10307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:35:17.002847 10307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:35:17.003846 10307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/data/instance:
uuid: "382a09288d7949cda5a027891aca35a4"
format_stamp: "Formatted at 2026-09-28 02:35:16 on dist-test-slave-fz47"
server_key: "43a2a754666dffb77d86859d6c2ca2dd"
server_key_iv: "23b08a390dd40a412c276099f4977208"
server_key_version: "encryptionkey@0"
I20260928 02:35:17.004158 10307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/wal/instance:
uuid: "382a09288d7949cda5a027891aca35a4"
format_stamp: "Formatted at 2026-09-28 02:35:16 on dist-test-slave-fz47"
server_key: "43a2a754666dffb77d86859d6c2ca2dd"
server_key_iv: "23b08a390dd40a412c276099f4977208"
server_key_version: "encryptionkey@0"
I20260928 02:35:17.005368 10307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:35:17.006108 10321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:17.006249 10307 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:35:17.006330 10307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "382a09288d7949cda5a027891aca35a4"
format_stamp: "Formatted at 2026-09-28 02:35:16 on dist-test-slave-fz47"
server_key: "43a2a754666dffb77d86859d6c2ca2dd"
server_key_iv: "23b08a390dd40a412c276099f4977208"
server_key_version: "encryptionkey@0"
I20260928 02:35:17.006392 10307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:35:17.009519 10307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:35:17.009737 10307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:35:17.009845 10307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:35:17.010025 10307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:35:17.010274 10307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:35:17.010304 10307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:17.010329 10307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:35:17.010351 10307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:17.015496 10307 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41963
I20260928 02:35:17.015529 10434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41963 every 8 connection(s)
I20260928 02:35:17.015797 10307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
I20260928 02:35:17.017658 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 10307
I20260928 02:35:17.017748 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-1/wal/instance
I20260928 02:35:17.017899 392 external_mini_cluster.cc:1428] Setting key 69888d7e4c47d59d57acafb7460688f7
I20260928 02:35:17.018416 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:17.016390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45571 (local address 127.0.98.62:42197)
0928 02:35:17.016432 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:17.016434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:17.016609 (+ 175us) server_negotiation.cc:407] Connection header received
0928 02:35:17.017045 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:17.017046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:17.017052 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:17.017099 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:17.018232 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:35:17.018233 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:35:17.018233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:35:17.018237 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:35:17.018252 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:35:17.018256 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:35:17.018297 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 02:35:17.018356 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 02:35:17.018367 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:35:17.018622 10437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:17.016369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42197 (local address 127.0.98.2:45571)
0928 02:35:17.016517 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:17.016547 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:17.016970 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:17.017149 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:17.017152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:17.017159 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:17.018102 (+ 943us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:35:17.018103 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:17.018129 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:35:17.018156 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:35:17.018299 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:35:17.018303 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:35:17.018305 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:35:17.018368 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 02:35:17.018424 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":74,"threads_started":1}
I20260928 02:35:17.018821 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42197
--builtin_ntp_servers=127.0.98.20:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 02:35:17.019129 10435 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42197
I20260928 02:35:17.019207 10435 heartbeater.cc:464] Registering TS with master...
I20260928 02:35:17.019333 10435 heartbeater.cc:511] Master 127.0.98.62:42197 requested a full tablet report, sending...
I20260928 02:35:17.019629 10116 ts_manager.cc:194] Registered new tserver with Master: 382a09288d7949cda5a027891aca35a4 (127.0.98.2:41963)
W20260928 02:35:17.095326 10438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:35:17.095528 10438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:35:17.095549 10438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:35:17.096704 10438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:35:17.096737 10438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:35:17.096764 10438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:35:17.098116 10438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:35:17.098328 10438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:35:17.098589 10438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:35:17.103093 10446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:35:17.103170 10443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:35:17.103223 10438 server_base.cc:1034] running on GCE node
W20260928 02:35:17.103098 10444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:35:17.103458 10438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:35:17.103673 10438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:35:17.104801 10438 hybrid_clock.cc:648] HybridClock initialized: now 1790562917104784 us; error 31 us; skew 500 ppm
I20260928 02:35:17.105922 10438 webserver.cc:460] Webserver started at http://127.0.98.3:43779/ using document root <none> and password file <none>
I20260928 02:35:17.106141 10438 fs_manager.cc:359] Metadata directory not provided
I20260928 02:35:17.106186 10438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:35:17.106307 10438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:35:17.107255 10438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/data/instance:
uuid: "e4482c205e2c4febad82965360714047"
format_stamp: "Formatted at 2026-09-28 02:35:17 on dist-test-slave-fz47"
server_key: "363a6df3136ee5e3e7161f275e261791"
server_key_iv: "c2a32f72289eb708d07b83b1e9b91b8e"
server_key_version: "encryptionkey@0"
I20260928 02:35:17.107558 10438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e4482c205e2c4febad82965360714047"
format_stamp: "Formatted at 2026-09-28 02:35:17 on dist-test-slave-fz47"
server_key: "363a6df3136ee5e3e7161f275e261791"
server_key_iv: "c2a32f72289eb708d07b83b1e9b91b8e"
server_key_version: "encryptionkey@0"
I20260928 02:35:17.108695 10438 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:35:17.109315 10452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:17.109444 10438 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:35:17.109520 10438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "e4482c205e2c4febad82965360714047"
format_stamp: "Formatted at 2026-09-28 02:35:17 on dist-test-slave-fz47"
server_key: "363a6df3136ee5e3e7161f275e261791"
server_key_iv: "c2a32f72289eb708d07b83b1e9b91b8e"
server_key_version: "encryptionkey@0"
I20260928 02:35:17.109583 10438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:35:17.112658 10438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:35:17.112877 10438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:35:17.112990 10438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:35:17.113157 10438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:35:17.113438 10438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:35:17.113471 10438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:17.113497 10438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:35:17.113520 10438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:17.118520 10438 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36997
I20260928 02:35:17.118572 10565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36997 every 8 connection(s)
I20260928 02:35:17.118821 10438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
I20260928 02:35:17.121353 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:17.119401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44167 (local address 127.0.98.62:42197)
0928 02:35:17.119459 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:17.119460 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:17.119661 (+ 201us) server_negotiation.cc:407] Connection header received
0928 02:35:17.120087 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:17.120088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:17.120095 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:17.120144 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:17.121112 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:35:17.121114 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:35:17.121114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:35:17.121118 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:35:17.121133 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:35:17.121138 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:35:17.121185 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 02:35:17.121272 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 02:35:17.121287 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 02:35:17.121505 10568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:17.119380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42197 (local address 127.0.98.3:44167)
0928 02:35:17.119556 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:17.119586 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:17.119998 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:17.120204 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:17.120208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:17.120215 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:17.120984 (+ 769us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:35:17.120984 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:17.121009 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:35:17.121034 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:35:17.121189 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:35:17.121193 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:35:17.121196 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:35:17.121274 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 02:35:17.121332 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260928 02:35:17.121884 10566 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42197
I20260928 02:35:17.121986 10566 heartbeater.cc:464] Registering TS with master...
I20260928 02:35:17.122114 10566 heartbeater.cc:511] Master 127.0.98.62:42197 requested a full tablet report, sending...
I20260928 02:35:17.122414 10116 ts_manager.cc:194] Registered new tserver with Master: e4482c205e2c4febad82965360714047 (127.0.98.3:36997)
I20260928 02:35:17.122877 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 10438
I20260928 02:35:17.122954 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790562862804661-392-0/minicluster-data/ts-2/wal/instance
I20260928 02:35:17.123093 392 external_mini_cluster.cc:1428] Setting key 1c1047d93944cfc9cd3c350d740c3dbb
I20260928 02:35:17.123754 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 02:35:17.125295 10156 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:35:17.124865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40904 (local address 127.0.98.62:42197)
0928 02:35:17.124935 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:17.124937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:17.125011 (+ 74us) server_negotiation.cc:407] Connection header received
0928 02:35:17.125072 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:17.125072 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:17.125077 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:17.125123 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:17.125220 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40904: BlockingRecv error: recv got EOF from 127.0.0.1:40904 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:35:17.125408 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 10176
I20260928 02:35:17.130330 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 10307
I20260928 02:35:17.133740 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 10438
I20260928 02:35:17.137127 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 10085
2026-09-28T02:35:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (492 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:35:17 dist-test-slave-fz47 krb5kdc[10576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:35:17 dist-test-slave-fz47 krb5kdc[10576](info): set up 2 sockets
Sep 28 02:35:17 dist-test-slave-fz47 krb5kdc[10576](info): commencing operation
krb5kdc: starting...
W20260928 02:35:18.085870 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:35:18 dist-test-slave-fz47 krb5kdc[10576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:35:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:35:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 02:35:18.191442 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44739
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44739
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:35:18.267015 10589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:35:18.267210 10589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:35:18.267237 10589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:35:18.268332 10589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:35:18.268368 10589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:35:18.268386 10589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:35:18.268401 10589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:35:18.268416 10589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:35:18.269850 10589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:35:18.270083 10589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:35:18.270269 10589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:35:18.274581 10595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:35:18.274631 10594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:35:18.274739 10589 server_base.cc:1034] running on GCE node
W20260928 02:35:18.274600 10597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:35:18.275040 10589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:35:18.275272 10589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:35:18.276405 10589 hybrid_clock.cc:648] HybridClock initialized: now 1790562918276398 us; error 32 us; skew 500 ppm
Sep 28 02:35:18 dist-test-slave-fz47 krb5kdc[10576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:35:18.282097 10589 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:35:18.282483 10589 webserver.cc:460] Webserver started at http://127.0.98.62:43545/ using document root <none> and password file <none>
I20260928 02:35:18.282673 10589 fs_manager.cc:359] Metadata directory not provided
I20260928 02:35:18.282719 10589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:35:18.282845 10589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:35:18.283880 10589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "908fd745e1804224bd953a61bcc474a1"
format_stamp: "Formatted at 2026-09-28 02:35:18 on dist-test-slave-fz47"
server_key: "cf95928f8bb423a02fa271bb391c0170"
server_key_iv: "8a89d0ca0ac70e6619071bed603c0324"
server_key_version: "encryptionkey@0"
I20260928 02:35:18.284194 10589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "908fd745e1804224bd953a61bcc474a1"
format_stamp: "Formatted at 2026-09-28 02:35:18 on dist-test-slave-fz47"
server_key: "cf95928f8bb423a02fa271bb391c0170"
server_key_iv: "8a89d0ca0ac70e6619071bed603c0324"
server_key_version: "encryptionkey@0"
I20260928 02:35:18.285415 10589 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:35:18.286038 10604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:18.286178 10589 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:35:18.286260 10589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "908fd745e1804224bd953a61bcc474a1"
format_stamp: "Formatted at 2026-09-28 02:35:18 on dist-test-slave-fz47"
server_key: "cf95928f8bb423a02fa271bb391c0170"
server_key_iv: "8a89d0ca0ac70e6619071bed603c0324"
server_key_version: "encryptionkey@0"
I20260928 02:35:18.286320 10589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:35:18.295245 10589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:35:18.296746 10589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:35:18.296869 10589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:35:18.300297 10589 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44739
I20260928 02:35:18.300324 10656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44739 every 8 connection(s)
I20260928 02:35:18.300609 10589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:35:18.301487 10657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:35:18.303468 10657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 908fd745e1804224bd953a61bcc474a1: Bootstrap starting.
I20260928 02:35:18.303969 10657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 908fd745e1804224bd953a61bcc474a1: Neither blocks nor log segments found. Creating new log.
I20260928 02:35:18.304217 10657 log.cc:824] T 00000000000000000000000000000000 P 908fd745e1804224bd953a61bcc474a1: Log is configured to *not* fsync() on all Append() calls
I20260928 02:35:18.304802 10657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 908fd745e1804224bd953a61bcc474a1: No bootstrap required, opened a new log
I20260928 02:35:18.305207 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 10589
I20260928 02:35:18.305322 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:35:18.305486 392 external_mini_cluster.cc:1428] Setting key e5bfb8a5a19e098a05885b9113362b5a
I20260928 02:35:18.306697 10657 raft_consensus.cc:348] T 00000000000000000000000000000000 P 908fd745e1804224bd953a61bcc474a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "908fd745e1804224bd953a61bcc474a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44739 } }
I20260928 02:35:18.306851 10657 raft_consensus.cc:374] T 00000000000000000000000000000000 P 908fd745e1804224bd953a61bcc474a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:35:18.306926 10657 raft_consensus.cc:729] T 00000000000000000000000000000000 P 908fd745e1804224bd953a61bcc474a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 908fd745e1804224bd953a61bcc474a1, State: Initialized, Role: FOLLOWER
I20260928 02:35:18.307072 10657 consensus_queue.cc:260] T 00000000000000000000000000000000 P 908fd745e1804224bd953a61bcc474a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "908fd745e1804224bd953a61bcc474a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44739 } }
I20260928 02:35:18.307150 10657 raft_consensus.cc:388] T 00000000000000000000000000000000 P 908fd745e1804224bd953a61bcc474a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:35:18.307206 10657 raft_consensus.cc:482] T 00000000000000000000000000000000 P 908fd745e1804224bd953a61bcc474a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:35:18.307252 10657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 908fd745e1804224bd953a61bcc474a1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:35:18.307826 10657 raft_consensus.cc:504] T 00000000000000000000000000000000 P 908fd745e1804224bd953a61bcc474a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "908fd745e1804224bd953a61bcc474a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44739 } }
I20260928 02:35:18.307933 10657 leader_election.cc:302] T 00000000000000000000000000000000 P 908fd745e1804224bd953a61bcc474a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 908fd745e1804224bd953a61bcc474a1; no voters:
I20260928 02:35:18.308073 10657 leader_election.cc:288] T 00000000000000000000000000000000 P 908fd745e1804224bd953a61bcc474a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:35:18.308111 10662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 908fd745e1804224bd953a61bcc474a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:35:18.308264 10662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 908fd745e1804224bd953a61bcc474a1 [term 1 LEADER]: Becoming Leader. State: Replica: 908fd745e1804224bd953a61bcc474a1, State: Running, Role: LEADER
I20260928 02:35:18.308277 10657 sys_catalog.cc:564] T 00000000000000000000000000000000 P 908fd745e1804224bd953a61bcc474a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:35:18.308422 10662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 908fd745e1804224bd953a61bcc474a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "908fd745e1804224bd953a61bcc474a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44739 } }
I20260928 02:35:18.308703 10662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 908fd745e1804224bd953a61bcc474a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 908fd745e1804224bd953a61bcc474a1. Latest consensus state: current_term: 1 leader_uuid: "908fd745e1804224bd953a61bcc474a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "908fd745e1804224bd953a61bcc474a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44739 } } }
I20260928 02:35:18.308917 10662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 908fd745e1804224bd953a61bcc474a1 [sys.catalog]: This master's current role is: LEADER
I20260928 02:35:18.309294 10670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:35:18.309751 10670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:35:18.308861 10664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 908fd745e1804224bd953a61bcc474a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "908fd745e1804224bd953a61bcc474a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "908fd745e1804224bd953a61bcc474a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44739 } } }
I20260928 02:35:18.310501 10664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 908fd745e1804224bd953a61bcc474a1 [sys.catalog]: This master's current role is: LEADER
I20260928 02:35:18.311441 10670 catalog_manager.cc:1347] Generated new cluster ID: 02fdca7cdd814663ba4a725ec565438f
I20260928 02:35:18.311493 10670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:35:18 dist-test-slave-fz47 krb5kdc[10576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:35:18.315369 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:18.305935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41932 (local address 127.0.98.62:44739)
0928 02:35:18.306110 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:18.306119 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:18.306151 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:35:18.306668 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:18.306676 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:18.306737 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:18.306878 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:18.307486 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:18.308749 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:18.309838 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:18.310317 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:18.310414 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:18.312629 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:35:18.312633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:35:18.312633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:35:18.312643 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:35:18.314083 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:35:18.314377 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:35:18.314380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:35:18.314381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:35:18.314472 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:35:18.314637 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:35:18.314638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:35:18.314639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:35:18.314855 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:35:18.314996 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 02:35:18.315083 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 02:35:18.315204 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":74,"threads_started":1}
I20260928 02:35:18.324049 10670 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:35:18.324510 10670 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:35:18.330181 10670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 908fd745e1804224bd953a61bcc474a1: Generated new TSK 0
I20260928 02:35:18.330322 10670 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 02:35:18.356475 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44739
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:35:18.449697 10683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:35:18.449887 10683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:35:18.449909 10683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:35:18.451038 10683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:35:18.451079 10683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:35:18.451103 10683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:35:18.452490 10683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:35:18.452699 10683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:35:18.452904 10683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:35:18.457410 10688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:35:18.457441 10691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:35:18.457487 10683 server_base.cc:1034] running on GCE node
W20260928 02:35:18.457499 10689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:35:18.457851 10683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:35:18.458037 10683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:35:18.459160 10683 hybrid_clock.cc:648] HybridClock initialized: now 1790562918459145 us; error 30 us; skew 500 ppm
Sep 28 02:35:18 dist-test-slave-fz47 krb5kdc[10576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:35:18.461413 10683 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 02:35:18.461758 10683 webserver.cc:460] Webserver started at http://127.0.98.1:44121/ using document root <none> and password file <none>
I20260928 02:35:18.461935 10683 fs_manager.cc:359] Metadata directory not provided
I20260928 02:35:18.461972 10683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:35:18.462057 10683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:35:18.462962 10683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-0/data/instance:
uuid: "76dd3d866ea24db992ed9a62fc09866f"
format_stamp: "Formatted at 2026-09-28 02:35:18 on dist-test-slave-fz47"
server_key: "27bcd4dbf7695f026087c2ed30334937"
server_key_iv: "6153bdfdc856c8af57220158536f7931"
server_key_version: "encryptionkey@0"
I20260928 02:35:18.463339 10683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-0/wal/instance:
uuid: "76dd3d866ea24db992ed9a62fc09866f"
format_stamp: "Formatted at 2026-09-28 02:35:18 on dist-test-slave-fz47"
server_key: "27bcd4dbf7695f026087c2ed30334937"
server_key_iv: "6153bdfdc856c8af57220158536f7931"
server_key_version: "encryptionkey@0"
I20260928 02:35:18.464437 10683 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:35:18.465027 10698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:18.465165 10683 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:18.465236 10683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "76dd3d866ea24db992ed9a62fc09866f"
format_stamp: "Formatted at 2026-09-28 02:35:18 on dist-test-slave-fz47"
server_key: "27bcd4dbf7695f026087c2ed30334937"
server_key_iv: "6153bdfdc856c8af57220158536f7931"
server_key_version: "encryptionkey@0"
I20260928 02:35:18.465319 10683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:35:18.478591 10683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:35:18.480125 10683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:35:18.480237 10683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:35:18.480432 10683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:35:18.480724 10683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:35:18.480773 10683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:18.480816 10683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:35:18.480839 10683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:18.486090 10683 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39257
I20260928 02:35:18.486141 10811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39257 every 8 connection(s)
I20260928 02:35:18.486406 10683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
I20260928 02:35:18.490826 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 10683
I20260928 02:35:18.490909 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-0/wal/instance
I20260928 02:35:18.491070 392 external_mini_cluster.cc:1428] Setting key 0d96fef1dd4375284aade8c71a19631d
Sep 28 02:35:18 dist-test-slave-fz47 krb5kdc[10576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:35:18.495019 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:18.487010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56855 (local address 127.0.98.62:44739)
0928 02:35:18.487058 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:18.487060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:18.487244 (+ 184us) server_negotiation.cc:407] Connection header received
0928 02:35:18.487669 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:18.487670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:18.487680 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:18.487732 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:18.488559 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:18.489079 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:18.490050 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:18.490248 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:18.490293 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:18.492482 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:35:18.492483 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:35:18.492483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:35:18.492491 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:35:18.493792 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:35:18.494175 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:35:18.494176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:35:18.494176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:35:18.494221 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:35:18.494453 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:35:18.494454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:35:18.494454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:35:18.494542 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:35:18.494582 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 02:35:18.494868 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 02:35:18.494928 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 02:35:18.495234 10814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:18.486989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44739 (local address 127.0.98.1:56855)
0928 02:35:18.487136 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:18.487174 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:18.487603 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:18.487782 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:18.487785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:18.488316 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:18.488494 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:18.488498 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:18.489155 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:18.489155 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:18.489979 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:18.489982 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:18.490299 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:18.490299 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:18.490383 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:18.490876 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:18.490885 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:18.492393 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:35:18.493849 (+ 1456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:35:18.493855 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:35:18.493857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:35:18.494112 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:35:18.494277 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:35:18.494278 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:35:18.494278 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:35:18.494401 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:35:18.494588 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:35:18.494592 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:35:18.494738 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 02:35:18.494847 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 02:35:18.495005 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
I20260928 02:35:18.495708 10812 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44739
I20260928 02:35:18.495800 10812 heartbeater.cc:464] Registering TS with master...
I20260928 02:35:18.495959 10812 heartbeater.cc:511] Master 127.0.98.62:44739 requested a full tablet report, sending...
I20260928 02:35:18.496492 10621 ts_manager.cc:194] Registered new tserver with Master: 76dd3d866ea24db992ed9a62fc09866f (127.0.98.1:39257)
I20260928 02:35:18.497414 10621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56855
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 02:35:18.512857 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44739
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:35:18.588630 10817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:35:18.588833 10817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:35:18.588863 10817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:35:18.590024 10817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:35:18.590062 10817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:35:18.590086 10817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:35:18.591456 10817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:35:18.591634 10817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:35:18.591816 10817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:35:18.596581 10822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:35:18.596577 10823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:35:18.596660 10825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:35:18.596606 10817 server_base.cc:1034] running on GCE node
I20260928 02:35:18.596933 10817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:35:18.597146 10817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:35:18.598271 10817 hybrid_clock.cc:648] HybridClock initialized: now 1790562918598255 us; error 31 us; skew 500 ppm
Sep 28 02:35:18 dist-test-slave-fz47 krb5kdc[10576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:35:18.600533 10817 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 02:35:18.600908 10817 webserver.cc:460] Webserver started at http://127.0.98.2:37489/ using document root <none> and password file <none>
I20260928 02:35:18.601099 10817 fs_manager.cc:359] Metadata directory not provided
I20260928 02:35:18.601143 10817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:35:18.601279 10817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:35:18.602237 10817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-1/data/instance:
uuid: "bc67fa0601014b97a2da1fc2adea0d42"
format_stamp: "Formatted at 2026-09-28 02:35:18 on dist-test-slave-fz47"
server_key: "b478ea83e7beb517447352efcd6a3b0e"
server_key_iv: "d324681255e2a21ecf71e894c5c7f920"
server_key_version: "encryptionkey@0"
I20260928 02:35:18.602545 10817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bc67fa0601014b97a2da1fc2adea0d42"
format_stamp: "Formatted at 2026-09-28 02:35:18 on dist-test-slave-fz47"
server_key: "b478ea83e7beb517447352efcd6a3b0e"
server_key_iv: "d324681255e2a21ecf71e894c5c7f920"
server_key_version: "encryptionkey@0"
I20260928 02:35:18.603667 10817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 02:35:18.604260 10832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:18.604421 10817 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:35:18.604496 10817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "bc67fa0601014b97a2da1fc2adea0d42"
format_stamp: "Formatted at 2026-09-28 02:35:18 on dist-test-slave-fz47"
server_key: "b478ea83e7beb517447352efcd6a3b0e"
server_key_iv: "d324681255e2a21ecf71e894c5c7f920"
server_key_version: "encryptionkey@0"
I20260928 02:35:18.604553 10817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:35:18.623617 10817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:35:18.625590 10817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:35:18.625715 10817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:35:18.625902 10817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:35:18.626173 10817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:35:18.626204 10817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:18.626230 10817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:35:18.626256 10817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:18.631415 10817 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40709
I20260928 02:35:18.631458 10945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40709 every 8 connection(s)
I20260928 02:35:18.631737 10817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
I20260928 02:35:18.636633 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 10817
I20260928 02:35:18.636721 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-1/wal/instance
I20260928 02:35:18.636904 392 external_mini_cluster.cc:1428] Setting key 9e52c0a9cd949f3d6e5978c5e7401124
Sep 28 02:35:18 dist-test-slave-fz47 krb5kdc[10576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:35:18.640093 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:18.632353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42983 (local address 127.0.98.62:44739)
0928 02:35:18.632429 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:18.632432 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:18.632574 (+ 142us) server_negotiation.cc:407] Connection header received
0928 02:35:18.633033 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:18.633034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:18.633044 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:18.633091 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:18.633976 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:18.634459 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:18.635355 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:18.635535 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:18.635583 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:18.637576 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:35:18.637577 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:35:18.637577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:35:18.637584 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:35:18.638881 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:35:18.639298 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:35:18.639298 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:35:18.639299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:35:18.639346 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:35:18.639565 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:35:18.639565 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:35:18.639566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:35:18.639651 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:35:18.639692 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 02:35:18.639976 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 02:35:18.640032 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:35:18.640312 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:18.632313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44739 (local address 127.0.98.2:42983)
0928 02:35:18.632464 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:18.632498 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:18.632961 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:18.633146 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:18.633149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:18.633715 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:18.633909 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:18.633914 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:18.634519 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:18.634520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:18.635292 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:18.635294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:18.635591 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:18.635592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:18.635691 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:18.636080 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:18.636087 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:18.637500 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:35:18.638958 (+ 1458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:35:18.638964 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:35:18.638966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:35:18.639236 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:35:18.639397 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:35:18.639398 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:35:18.639399 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:35:18.639516 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:35:18.639702 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:35:18.639707 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:35:18.639859 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 02:35:18.639956 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 02:35:18.640096 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":75,"threads_started":1}
I20260928 02:35:18.640736 10946 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44739
I20260928 02:35:18.640813 10946 heartbeater.cc:464] Registering TS with master...
I20260928 02:35:18.640985 10946 heartbeater.cc:511] Master 127.0.98.62:44739 requested a full tablet report, sending...
I20260928 02:35:18.641362 10621 ts_manager.cc:194] Registered new tserver with Master: bc67fa0601014b97a2da1fc2adea0d42 (127.0.98.2:40709)
I20260928 02:35:18.641702 10621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42983
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 02:35:18.659691 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44739
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:35:18.735795 10951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:35:18.735970 10951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:35:18.735991 10951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:35:18.737071 10951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:35:18.737107 10951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:35:18.737134 10951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:35:18.738547 10951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:35:18.738744 10951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:35:18.738936 10951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:35:18.743428 10957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:35:18.743435 10959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:35:18.743484 10956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:35:18.743628 10951 server_base.cc:1034] running on GCE node
I20260928 02:35:18.743860 10951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:35:18.744069 10951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:35:18.745205 10951 hybrid_clock.cc:648] HybridClock initialized: now 1790562918745190 us; error 31 us; skew 500 ppm
Sep 28 02:35:18 dist-test-slave-fz47 krb5kdc[10576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:35:18.747401 10951 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 02:35:18.747790 10951 webserver.cc:460] Webserver started at http://127.0.98.3:36211/ using document root <none> and password file <none>
I20260928 02:35:18.747985 10951 fs_manager.cc:359] Metadata directory not provided
I20260928 02:35:18.748030 10951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:35:18.748142 10951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:35:18.749074 10951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-2/data/instance:
uuid: "817f6cdc7e2441d580c08e62681c6160"
format_stamp: "Formatted at 2026-09-28 02:35:18 on dist-test-slave-fz47"
server_key: "3b5208d46dc84aa5052d51bbcb44231c"
server_key_iv: "5dc0f45a3778b3711e2b2292c07c06dd"
server_key_version: "encryptionkey@0"
I20260928 02:35:18.749428 10951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-2/wal/instance:
uuid: "817f6cdc7e2441d580c08e62681c6160"
format_stamp: "Formatted at 2026-09-28 02:35:18 on dist-test-slave-fz47"
server_key: "3b5208d46dc84aa5052d51bbcb44231c"
server_key_iv: "5dc0f45a3778b3711e2b2292c07c06dd"
server_key_version: "encryptionkey@0"
I20260928 02:35:18.750557 10951 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:35:18.751200 10966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:18.751368 10951 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:35:18.751436 10951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "817f6cdc7e2441d580c08e62681c6160"
format_stamp: "Formatted at 2026-09-28 02:35:18 on dist-test-slave-fz47"
server_key: "3b5208d46dc84aa5052d51bbcb44231c"
server_key_iv: "5dc0f45a3778b3711e2b2292c07c06dd"
server_key_version: "encryptionkey@0"
I20260928 02:35:18.751494 10951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:35:18.771930 10951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:35:18.773475 10951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:35:18.773593 10951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:35:18.773761 10951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:35:18.774005 10951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:35:18.774036 10951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:18.774061 10951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:35:18.774085 10951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:35:18.779229 10951 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33855
I20260928 02:35:18.779277 11079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33855 every 8 connection(s)
I20260928 02:35:18.779537 10951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
I20260928 02:35:18.783721 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 10951
I20260928 02:35:18.783802 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-2/wal/instance
I20260928 02:35:18.783962 392 external_mini_cluster.cc:1428] Setting key 117822fe47e2608f2f077b91e16e0936
Sep 28 02:35:18 dist-test-slave-fz47 krb5kdc[10576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:35:18.787711 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:18.780120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39685 (local address 127.0.98.62:44739)
0928 02:35:18.780160 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:18.780162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:18.780325 (+ 163us) server_negotiation.cc:407] Connection header received
0928 02:35:18.780804 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:18.780806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:18.780816 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:18.780864 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:18.781764 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:18.782240 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:18.783169 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:18.783361 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:18.783413 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:18.785485 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:35:18.785486 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:35:18.785486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:35:18.785493 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:35:18.786565 (+ 1072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:35:18.786905 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:35:18.786906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:35:18.786906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:35:18.786957 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:35:18.787160 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:35:18.787161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:35:18.787161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:35:18.787263 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:35:18.787324 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 02:35:18.787553 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 02:35:18.787625 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:35:18.787863 11082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:18.780084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44739 (local address 127.0.98.3:39685)
0928 02:35:18.780212 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:18.780251 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:18.780714 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:18.780927 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:18.780930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:18.781499 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:35:18.781671 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:18.781675 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:18.782316 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:18.782317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:18.783086 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:18.783088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:18.783425 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:18.783426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:18.783536 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:18.783975 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:35:18.783983 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:35:18.785405 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:35:18.786632 (+ 1227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:35:18.786637 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:35:18.786639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:35:18.786841 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:35:18.787009 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:35:18.787010 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:35:18.787010 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:35:18.787107 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:35:18.787316 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:35:18.787321 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:35:18.787445 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 02:35:18.787524 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 02:35:18.787658 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":61,"threads_started":1}
I20260928 02:35:18.788251 11080 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44739
I20260928 02:35:18.788327 11080 heartbeater.cc:464] Registering TS with master...
I20260928 02:35:18.788450 11080 heartbeater.cc:511] Master 127.0.98.62:44739 requested a full tablet report, sending...
I20260928 02:35:18.788738 10621 ts_manager.cc:194] Registered new tserver with Master: 817f6cdc7e2441d580c08e62681c6160 (127.0.98.3:33855)
I20260928 02:35:18.789054 10621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39685
I20260928 02:35:18.795354 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:35:18.801994 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:18.796541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41942 (local address 127.0.98.62:44739)
0928 02:35:18.796598 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:18.796600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:18.796683 (+ 83us) server_negotiation.cc:407] Connection header received
0928 02:35:18.796744 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:18.796745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:18.796755 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:18.796800 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:35:18.797359 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:18.797873 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:18.798567 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:18.798757 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:18.798808 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:18.799870 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:35:18.799871 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:35:18.799872 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:35:18.799878 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:35:18.800979 (+ 1101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:35:18.801280 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:35:18.801281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:35:18.801282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:35:18.801335 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:35:18.801494 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:35:18.801495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:35:18.801495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:35:18.801630 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:35:18.801713 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 02:35:18.801839 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 02:35:18.801909 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 02:35:18.803285 10621 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:35:18.804747 10621 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:35:18.810563 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:18.806658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33855 (local address 127.0.0.1:33006)
0928 02:35:18.807011 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:18.807021 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:18.807113 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:18.807363 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:18.807364 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:18.807365 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:35:18.807408 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:18.807409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:18.808715 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:18.808716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:18.809657 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:18.809659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:18.810370 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:18.810371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:18.810423 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:18.810433 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 02:35:18.810480 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 02:35:18.810515 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":8,"thread_start_us":53,"threads_started":1}
I20260928 02:35:18.810685 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:18.806740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33006 (local address 127.0.98.3:33855)
0928 02:35:18.806903 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:18.806908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:18.807082 (+ 174us) server_negotiation.cc:407] Connection header received
0928 02:35:18.807210 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:18.807213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:18.807227 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:18.807292 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:35:18.807481 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:18.808638 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:18.809722 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:18.810302 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:18.810373 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:18.810429 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:35:18.810500 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 02:35:18.810533 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260928 02:35:18.811116 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:18.806850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39257 (local address 127.0.0.1:43370)
0928 02:35:18.807208 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:18.807215 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:18.807279 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:18.807496 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:18.807497 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:18.807499 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:35:18.807548 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:18.807549 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:18.808550 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:18.808551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:18.809948 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:18.809950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:18.810903 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:18.810904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:18.810967 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:18.810972 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:35:18.811016 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 02:35:18.811051 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":26,"thread_start_us":38,"threads_started":1}
I20260928 02:35:18.811319 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:18.807056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43370 (local address 127.0.98.1:39257)
0928 02:35:18.807229 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:18.807233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:18.807256 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:35:18.807348 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:18.807351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:18.807365 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:18.807425 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:35:18.807617 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:18.808417 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:18.810141 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:18.810833 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:18.810894 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:18.810947 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:35:18.811208 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 02:35:18.811246 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":72,"threads_started":1}
I20260928 02:35:18.811302 11014 tablet_service.cc:1467] Processing CreateTablet for tablet 43c45b848adc46b0806f8b3316bc7479 (DEFAULT_TABLE table=test-table [id=bad0831a189b4e4f99cb90fd7c6dde70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:35:18.811630 11014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43c45b848adc46b0806f8b3316bc7479. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:35:18.811966 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:18.806658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40709 (local address 127.0.0.1:44980)
0928 02:35:18.806931 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:18.807011 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:18.807080 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:18.807310 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:18.807312 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:18.807315 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:35:18.807408 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:18.807409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:18.808651 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:18.808652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:18.809940 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:18.809943 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:18.811739 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:18.811740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:18.811809 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:18.811813 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:35:18.811867 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 02:35:18.811904 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":29,"spinlock_wait_cycles":768,"thread_start_us":58,"threads_started":1}
I20260928 02:35:18.812165 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:18.806855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44980 (local address 127.0.98.2:40709)
0928 02:35:18.806993 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:18.806996 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:18.807059 (+ 63us) server_negotiation.cc:407] Connection header received
0928 02:35:18.807149 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:18.807151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:18.807165 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:18.807227 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:35:18.807481 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:18.808563 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:18.810978 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:18.811663 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:18.811727 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:18.811780 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:35:18.812055 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 02:35:18.812093 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":76,"threads_started":1}
I20260928 02:35:18.813952 10880 tablet_service.cc:1467] Processing CreateTablet for tablet 43c45b848adc46b0806f8b3316bc7479 (DEFAULT_TABLE table=test-table [id=bad0831a189b4e4f99cb90fd7c6dde70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:35:18.813928 10746 tablet_service.cc:1467] Processing CreateTablet for tablet 43c45b848adc46b0806f8b3316bc7479 (DEFAULT_TABLE table=test-table [id=bad0831a189b4e4f99cb90fd7c6dde70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:35:18.814239 10746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43c45b848adc46b0806f8b3316bc7479. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:35:18.814239 10880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43c45b848adc46b0806f8b3316bc7479. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:35:18.814234 11094 tablet_bootstrap.cc:492] T 43c45b848adc46b0806f8b3316bc7479 P 817f6cdc7e2441d580c08e62681c6160: Bootstrap starting.
I20260928 02:35:18.814862 11094 tablet_bootstrap.cc:654] T 43c45b848adc46b0806f8b3316bc7479 P 817f6cdc7e2441d580c08e62681c6160: Neither blocks nor log segments found. Creating new log.
I20260928 02:35:18.815402 11094 log.cc:824] T 43c45b848adc46b0806f8b3316bc7479 P 817f6cdc7e2441d580c08e62681c6160: Log is configured to *not* fsync() on all Append() calls
I20260928 02:35:18.816203 11094 tablet_bootstrap.cc:492] T 43c45b848adc46b0806f8b3316bc7479 P 817f6cdc7e2441d580c08e62681c6160: No bootstrap required, opened a new log
I20260928 02:35:18.816267 11094 ts_tablet_manager.cc:1397] T 43c45b848adc46b0806f8b3316bc7479 P 817f6cdc7e2441d580c08e62681c6160: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:35:18.817224 11097 tablet_bootstrap.cc:492] T 43c45b848adc46b0806f8b3316bc7479 P 76dd3d866ea24db992ed9a62fc09866f: Bootstrap starting.
I20260928 02:35:18.817869 11097 tablet_bootstrap.cc:654] T 43c45b848adc46b0806f8b3316bc7479 P 76dd3d866ea24db992ed9a62fc09866f: Neither blocks nor log segments found. Creating new log.
I20260928 02:35:18.817736 11094 raft_consensus.cc:348] T 43c45b848adc46b0806f8b3316bc7479 P 817f6cdc7e2441d580c08e62681c6160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc67fa0601014b97a2da1fc2adea0d42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40709 } } peers { permanent_uuid: "817f6cdc7e2441d580c08e62681c6160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 } } peers { permanent_uuid: "76dd3d866ea24db992ed9a62fc09866f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39257 } }
I20260928 02:35:18.817914 11094 raft_consensus.cc:374] T 43c45b848adc46b0806f8b3316bc7479 P 817f6cdc7e2441d580c08e62681c6160 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:35:18.817946 11094 raft_consensus.cc:729] T 43c45b848adc46b0806f8b3316bc7479 P 817f6cdc7e2441d580c08e62681c6160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 817f6cdc7e2441d580c08e62681c6160, State: Initialized, Role: FOLLOWER
I20260928 02:35:18.818073 11094 consensus_queue.cc:260] T 43c45b848adc46b0806f8b3316bc7479 P 817f6cdc7e2441d580c08e62681c6160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc67fa0601014b97a2da1fc2adea0d42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40709 } } peers { permanent_uuid: "817f6cdc7e2441d580c08e62681c6160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 } } peers { permanent_uuid: "76dd3d866ea24db992ed9a62fc09866f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39257 } }
I20260928 02:35:18.818338 11094 ts_tablet_manager.cc:1428] T 43c45b848adc46b0806f8b3316bc7479 P 817f6cdc7e2441d580c08e62681c6160: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:35:18.818349 11080 heartbeater.cc:503] Master 127.0.98.62:44739 was elected leader, sending a full tablet report...
I20260928 02:35:18.818382 11097 log.cc:824] T 43c45b848adc46b0806f8b3316bc7479 P 76dd3d866ea24db992ed9a62fc09866f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:35:18.818681 11096 tablet_bootstrap.cc:492] T 43c45b848adc46b0806f8b3316bc7479 P bc67fa0601014b97a2da1fc2adea0d42: Bootstrap starting.
I20260928 02:35:18.819242 11096 tablet_bootstrap.cc:654] T 43c45b848adc46b0806f8b3316bc7479 P bc67fa0601014b97a2da1fc2adea0d42: Neither blocks nor log segments found. Creating new log.
I20260928 02:35:18.819653 11096 log.cc:824] T 43c45b848adc46b0806f8b3316bc7479 P bc67fa0601014b97a2da1fc2adea0d42: Log is configured to *not* fsync() on all Append() calls
I20260928 02:35:18.820048 11097 tablet_bootstrap.cc:492] T 43c45b848adc46b0806f8b3316bc7479 P 76dd3d866ea24db992ed9a62fc09866f: No bootstrap required, opened a new log
I20260928 02:35:18.820102 11097 ts_tablet_manager.cc:1397] T 43c45b848adc46b0806f8b3316bc7479 P 76dd3d866ea24db992ed9a62fc09866f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 02:35:18.820253 11096 tablet_bootstrap.cc:492] T 43c45b848adc46b0806f8b3316bc7479 P bc67fa0601014b97a2da1fc2adea0d42: No bootstrap required, opened a new log
I20260928 02:35:18.820304 11096 ts_tablet_manager.cc:1397] T 43c45b848adc46b0806f8b3316bc7479 P bc67fa0601014b97a2da1fc2adea0d42: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:35:18.821476 11097 raft_consensus.cc:348] T 43c45b848adc46b0806f8b3316bc7479 P 76dd3d866ea24db992ed9a62fc09866f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc67fa0601014b97a2da1fc2adea0d42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40709 } } peers { permanent_uuid: "817f6cdc7e2441d580c08e62681c6160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 } } peers { permanent_uuid: "76dd3d866ea24db992ed9a62fc09866f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39257 } }
I20260928 02:35:18.821606 11097 raft_consensus.cc:374] T 43c45b848adc46b0806f8b3316bc7479 P 76dd3d866ea24db992ed9a62fc09866f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:35:18.821635 11097 raft_consensus.cc:729] T 43c45b848adc46b0806f8b3316bc7479 P 76dd3d866ea24db992ed9a62fc09866f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76dd3d866ea24db992ed9a62fc09866f, State: Initialized, Role: FOLLOWER
I20260928 02:35:18.821738 11097 consensus_queue.cc:260] T 43c45b848adc46b0806f8b3316bc7479 P 76dd3d866ea24db992ed9a62fc09866f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc67fa0601014b97a2da1fc2adea0d42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40709 } } peers { permanent_uuid: "817f6cdc7e2441d580c08e62681c6160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 } } peers { permanent_uuid: "76dd3d866ea24db992ed9a62fc09866f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39257 } }
I20260928 02:35:18.821794 11096 raft_consensus.cc:348] T 43c45b848adc46b0806f8b3316bc7479 P bc67fa0601014b97a2da1fc2adea0d42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc67fa0601014b97a2da1fc2adea0d42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40709 } } peers { permanent_uuid: "817f6cdc7e2441d580c08e62681c6160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 } } peers { permanent_uuid: "76dd3d866ea24db992ed9a62fc09866f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39257 } }
I20260928 02:35:18.821941 11096 raft_consensus.cc:374] T 43c45b848adc46b0806f8b3316bc7479 P bc67fa0601014b97a2da1fc2adea0d42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:35:18.821978 11096 raft_consensus.cc:729] T 43c45b848adc46b0806f8b3316bc7479 P bc67fa0601014b97a2da1fc2adea0d42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc67fa0601014b97a2da1fc2adea0d42, State: Initialized, Role: FOLLOWER
I20260928 02:35:18.821979 10812 heartbeater.cc:503] Master 127.0.98.62:44739 was elected leader, sending a full tablet report...
I20260928 02:35:18.822083 11096 consensus_queue.cc:260] T 43c45b848adc46b0806f8b3316bc7479 P bc67fa0601014b97a2da1fc2adea0d42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc67fa0601014b97a2da1fc2adea0d42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40709 } } peers { permanent_uuid: "817f6cdc7e2441d580c08e62681c6160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 } } peers { permanent_uuid: "76dd3d866ea24db992ed9a62fc09866f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39257 } }
I20260928 02:35:18.821971 11097 ts_tablet_manager.cc:1428] T 43c45b848adc46b0806f8b3316bc7479 P 76dd3d866ea24db992ed9a62fc09866f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 02:35:18.822314 11096 ts_tablet_manager.cc:1428] T 43c45b848adc46b0806f8b3316bc7479 P bc67fa0601014b97a2da1fc2adea0d42: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:35:18.822335 10946 heartbeater.cc:503] Master 127.0.98.62:44739 was elected leader, sending a full tablet report...
W20260928 02:35:18.882203 10947 tablet.cc:2369] T 43c45b848adc46b0806f8b3316bc7479 P bc67fa0601014b97a2da1fc2adea0d42: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:35:18.986997 10813 tablet.cc:2369] T 43c45b848adc46b0806f8b3316bc7479 P 76dd3d866ea24db992ed9a62fc09866f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:35:19.005497 11101 raft_consensus.cc:482] T 43c45b848adc46b0806f8b3316bc7479 P 76dd3d866ea24db992ed9a62fc09866f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:35:19.005616 11101 raft_consensus.cc:504] T 43c45b848adc46b0806f8b3316bc7479 P 76dd3d866ea24db992ed9a62fc09866f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc67fa0601014b97a2da1fc2adea0d42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40709 } } peers { permanent_uuid: "817f6cdc7e2441d580c08e62681c6160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 } } peers { permanent_uuid: "76dd3d866ea24db992ed9a62fc09866f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39257 } }
I20260928 02:35:19.005941 11101 leader_election.cc:288] T 43c45b848adc46b0806f8b3316bc7479 P 76dd3d866ea24db992ed9a62fc09866f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc67fa0601014b97a2da1fc2adea0d42 (127.0.98.2:40709), 817f6cdc7e2441d580c08e62681c6160 (127.0.98.3:33855)
I20260928 02:35:19.008790 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:19.006182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40401 (local address 127.0.98.2:40709)
0928 02:35:19.006228 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:19.006230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:19.006335 (+ 105us) server_negotiation.cc:407] Connection header received
0928 02:35:19.006385 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:19.006386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:19.006397 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:19.006443 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:35:19.006582 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:19.007036 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:19.008112 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:19.008479 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:19.008537 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:19.008547 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 02:35:19.008671 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 02:35:19.008705 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 02:35:19.008775 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:19.006171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45851 (local address 127.0.98.3:33855)
0928 02:35:19.006213 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:19.006215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:19.006290 (+ 75us) server_negotiation.cc:407] Connection header received
0928 02:35:19.006363 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:19.006364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:19.006375 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:19.006421 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:35:19.006549 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:19.007045 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:19.008114 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:19.008483 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:19.008541 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:19.008550 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:35:19.008674 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 02:35:19.008705 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:35:19.008844 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:19.006100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33855 (local address 127.0.98.1:45851)
0928 02:35:19.006229 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:19.006239 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:19.006296 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:19.006425 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:19.006426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:19.006428 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:35:19.006495 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:19.006497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:19.007113 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:19.007114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:19.008048 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:19.008050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:19.008550 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:19.008551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:19.008618 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:19.008621 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:35:19.008665 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 02:35:19.008707 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":19,"thread_start_us":61,"threads_started":1}
I20260928 02:35:19.008847 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:19.006099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40709 (local address 127.0.98.1:40401)
0928 02:35:19.006272 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:19.006281 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:35:19.006335 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:35:19.006447 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:35:19.006448 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:35:19.006449 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:35:19.006529 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:19.006530 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:19.007105 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:19.007106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:19.008037 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:35:19.008039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:35:19.008547 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:35:19.008548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:35:19.008617 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:19.008620 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:35:19.008662 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 02:35:19.008705 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":42,"spinlock_wait_cycles":11136,"thread_start_us":97,"threads_started":1}
I20260928 02:35:19.009196 10900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43c45b848adc46b0806f8b3316bc7479" candidate_uuid: "76dd3d866ea24db992ed9a62fc09866f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc67fa0601014b97a2da1fc2adea0d42" is_pre_election: true
I20260928 02:35:19.009194 11034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43c45b848adc46b0806f8b3316bc7479" candidate_uuid: "76dd3d866ea24db992ed9a62fc09866f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "817f6cdc7e2441d580c08e62681c6160" is_pre_election: true
I20260928 02:35:19.009361 10900 raft_consensus.cc:2454] T 43c45b848adc46b0806f8b3316bc7479 P bc67fa0601014b97a2da1fc2adea0d42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76dd3d866ea24db992ed9a62fc09866f in term 0.
I20260928 02:35:19.009361 11034 raft_consensus.cc:2454] T 43c45b848adc46b0806f8b3316bc7479 P 817f6cdc7e2441d580c08e62681c6160 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76dd3d866ea24db992ed9a62fc09866f in term 0.
I20260928 02:35:19.009555 10701 leader_election.cc:302] T 43c45b848adc46b0806f8b3316bc7479 P 76dd3d866ea24db992ed9a62fc09866f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76dd3d866ea24db992ed9a62fc09866f, bc67fa0601014b97a2da1fc2adea0d42; no voters:
I20260928 02:35:19.009694 11101 raft_consensus.cc:2781] T 43c45b848adc46b0806f8b3316bc7479 P 76dd3d866ea24db992ed9a62fc09866f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:35:19.009730 11101 raft_consensus.cc:482] T 43c45b848adc46b0806f8b3316bc7479 P 76dd3d866ea24db992ed9a62fc09866f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:35:19.009747 11101 raft_consensus.cc:3037] T 43c45b848adc46b0806f8b3316bc7479 P 76dd3d866ea24db992ed9a62fc09866f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:35:19.010437 11101 raft_consensus.cc:504] T 43c45b848adc46b0806f8b3316bc7479 P 76dd3d866ea24db992ed9a62fc09866f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc67fa0601014b97a2da1fc2adea0d42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40709 } } peers { permanent_uuid: "817f6cdc7e2441d580c08e62681c6160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 } } peers { permanent_uuid: "76dd3d866ea24db992ed9a62fc09866f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39257 } }
I20260928 02:35:19.010545 11101 leader_election.cc:288] T 43c45b848adc46b0806f8b3316bc7479 P 76dd3d866ea24db992ed9a62fc09866f [CANDIDATE]: Term 1 election: Requested vote from peers bc67fa0601014b97a2da1fc2adea0d42 (127.0.98.2:40709), 817f6cdc7e2441d580c08e62681c6160 (127.0.98.3:33855)
I20260928 02:35:19.010669 11034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43c45b848adc46b0806f8b3316bc7479" candidate_uuid: "76dd3d866ea24db992ed9a62fc09866f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "817f6cdc7e2441d580c08e62681c6160"
I20260928 02:35:19.010660 10900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43c45b848adc46b0806f8b3316bc7479" candidate_uuid: "76dd3d866ea24db992ed9a62fc09866f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc67fa0601014b97a2da1fc2adea0d42"
I20260928 02:35:19.010730 11034 raft_consensus.cc:3037] T 43c45b848adc46b0806f8b3316bc7479 P 817f6cdc7e2441d580c08e62681c6160 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:35:19.010738 10900 raft_consensus.cc:3037] T 43c45b848adc46b0806f8b3316bc7479 P bc67fa0601014b97a2da1fc2adea0d42 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:35:19.011353 11034 raft_consensus.cc:2454] T 43c45b848adc46b0806f8b3316bc7479 P 817f6cdc7e2441d580c08e62681c6160 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76dd3d866ea24db992ed9a62fc09866f in term 1.
I20260928 02:35:19.011431 10900 raft_consensus.cc:2454] T 43c45b848adc46b0806f8b3316bc7479 P bc67fa0601014b97a2da1fc2adea0d42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76dd3d866ea24db992ed9a62fc09866f in term 1.
I20260928 02:35:19.011511 10699 leader_election.cc:302] T 43c45b848adc46b0806f8b3316bc7479 P 76dd3d866ea24db992ed9a62fc09866f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76dd3d866ea24db992ed9a62fc09866f, 817f6cdc7e2441d580c08e62681c6160; no voters:
I20260928 02:35:19.011590 11101 raft_consensus.cc:2781] T 43c45b848adc46b0806f8b3316bc7479 P 76dd3d866ea24db992ed9a62fc09866f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:35:19.011700 11101 raft_consensus.cc:686] T 43c45b848adc46b0806f8b3316bc7479 P 76dd3d866ea24db992ed9a62fc09866f [term 1 LEADER]: Becoming Leader. State: Replica: 76dd3d866ea24db992ed9a62fc09866f, State: Running, Role: LEADER
I20260928 02:35:19.011771 11101 consensus_queue.cc:237] T 43c45b848adc46b0806f8b3316bc7479 P 76dd3d866ea24db992ed9a62fc09866f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc67fa0601014b97a2da1fc2adea0d42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40709 } } peers { permanent_uuid: "817f6cdc7e2441d580c08e62681c6160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 } } peers { permanent_uuid: "76dd3d866ea24db992ed9a62fc09866f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39257 } }
I20260928 02:35:19.012493 10620 catalog_manager.cc:5495] T 43c45b848adc46b0806f8b3316bc7479 P 76dd3d866ea24db992ed9a62fc09866f reported cstate change: term changed from 0 to 1, leader changed from <none> to 76dd3d866ea24db992ed9a62fc09866f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "76dd3d866ea24db992ed9a62fc09866f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc67fa0601014b97a2da1fc2adea0d42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "817f6cdc7e2441d580c08e62681c6160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76dd3d866ea24db992ed9a62fc09866f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39257 } health_report { overall_health: HEALTHY } } }
W20260928 02:35:19.029971 11081 tablet.cc:2369] T 43c45b848adc46b0806f8b3316bc7479 P 817f6cdc7e2441d580c08e62681c6160: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:35:19.036623 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:19.034516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43386 (local address 127.0.98.1:39257)
0928 02:35:19.034574 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 02:35:19.034576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:35:19.034583 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:35:19.034613 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:35:19.034613 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:35:19.034630 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:35:19.034685 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:35:19.034810 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:19.035243 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:19.035970 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:35:19.036144 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:35:19.036197 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:35:19.036348 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:35:19.036437 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:35:19.036503 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:35:19.036539 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 02:35:19.036557 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 02:35:19.038111 10900 raft_consensus.cc:1261] T 43c45b848adc46b0806f8b3316bc7479 P bc67fa0601014b97a2da1fc2adea0d42 [term 1 FOLLOWER]: Refusing update from remote peer 76dd3d866ea24db992ed9a62fc09866f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:35:19.038370 11034 raft_consensus.cc:1261] T 43c45b848adc46b0806f8b3316bc7479 P 817f6cdc7e2441d580c08e62681c6160 [term 1 FOLLOWER]: Refusing update from remote peer 76dd3d866ea24db992ed9a62fc09866f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:35:19.038409 11101 consensus_queue.cc:1035] T 43c45b848adc46b0806f8b3316bc7479 P 76dd3d866ea24db992ed9a62fc09866f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc67fa0601014b97a2da1fc2adea0d42" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:35:19.038600 11105 consensus_queue.cc:1035] T 43c45b848adc46b0806f8b3316bc7479 P 76dd3d866ea24db992ed9a62fc09866f [LEADER]: Connected to new peer: Peer: permanent_uuid: "817f6cdc7e2441d580c08e62681c6160" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:35:19.040432 11108 mvcc.cc:205] Tried to move back new op lower bound from 7334145716377980928 to 7334145716272775168. Current Snapshot: MvccSnapshot[applied={T|T < 7334145716377980928}]
I20260928 02:35:19.040593 11109 mvcc.cc:205] Tried to move back new op lower bound from 7334145716377980928 to 7334145716272775168. Current Snapshot: MvccSnapshot[applied={T|T < 7334145716377980928}]
I20260928 02:35:19.041006 11110 mvcc.cc:205] Tried to move back new op lower bound from 7334145716377980928 to 7334145716272775168. Current Snapshot: MvccSnapshot[applied={T|T < 7334145716377980928}]
I20260928 02:35:19.043020 10621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:35:19.043114 10621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:35:19.043762 10621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 908fd745e1804224bd953a61bcc474a1: Sending DeleteTablet for 3 replicas of tablet 43c45b848adc46b0806f8b3316bc7479
I20260928 02:35:19.044062 11014 tablet_service.cc:1514] Processing DeleteTablet for tablet 43c45b848adc46b0806f8b3316bc7479 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:35:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33006
I20260928 02:35:19.044080 10880 tablet_service.cc:1514] Processing DeleteTablet for tablet 43c45b848adc46b0806f8b3316bc7479 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:35:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44980
I20260928 02:35:19.044281 11119 tablet_replica.cc:330] T 43c45b848adc46b0806f8b3316bc7479 P bc67fa0601014b97a2da1fc2adea0d42: stopping tablet replica
I20260928 02:35:19.044269 10746 tablet_service.cc:1514] Processing DeleteTablet for tablet 43c45b848adc46b0806f8b3316bc7479 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:35:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43370
I20260928 02:35:19.044353 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 10683
I20260928 02:35:19.044373 11119 raft_consensus.cc:2229] T 43c45b848adc46b0806f8b3316bc7479 P bc67fa0601014b97a2da1fc2adea0d42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:35:19.044461 11119 raft_consensus.cc:2258] T 43c45b848adc46b0806f8b3316bc7479 P bc67fa0601014b97a2da1fc2adea0d42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:35:19.044824 11118 tablet_replica.cc:330] T 43c45b848adc46b0806f8b3316bc7479 P 817f6cdc7e2441d580c08e62681c6160: stopping tablet replica
I20260928 02:35:19.044902 11118 raft_consensus.cc:2229] T 43c45b848adc46b0806f8b3316bc7479 P 817f6cdc7e2441d580c08e62681c6160 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:35:19.044981 11118 raft_consensus.cc:2258] T 43c45b848adc46b0806f8b3316bc7479 P 817f6cdc7e2441d580c08e62681c6160 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:35:19.045239 11119 ts_tablet_manager.cc:1905] T 43c45b848adc46b0806f8b3316bc7479 P bc67fa0601014b97a2da1fc2adea0d42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:35:19.045291 11118 ts_tablet_manager.cc:1905] T 43c45b848adc46b0806f8b3316bc7479 P 817f6cdc7e2441d580c08e62681c6160: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:35:19.046960 11119 ts_tablet_manager.cc:1918] T 43c45b848adc46b0806f8b3316bc7479 P bc67fa0601014b97a2da1fc2adea0d42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:35:19.046960 11118 ts_tablet_manager.cc:1918] T 43c45b848adc46b0806f8b3316bc7479 P 817f6cdc7e2441d580c08e62681c6160: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:35:19.047031 11119 log.cc:1196] T 43c45b848adc46b0806f8b3316bc7479 P bc67fa0601014b97a2da1fc2adea0d42: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-1/wal/wals/43c45b848adc46b0806f8b3316bc7479
I20260928 02:35:19.047031 11118 log.cc:1196] T 43c45b848adc46b0806f8b3316bc7479 P 817f6cdc7e2441d580c08e62681c6160: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790562862804661-392-0/minicluster-data/ts-2/wal/wals/43c45b848adc46b0806f8b3316bc7479
I20260928 02:35:19.047289 11119 ts_tablet_manager.cc:1939] T 43c45b848adc46b0806f8b3316bc7479 P bc67fa0601014b97a2da1fc2adea0d42: Deleting consensus metadata
I20260928 02:35:19.047310 11118 ts_tablet_manager.cc:1939] T 43c45b848adc46b0806f8b3316bc7479 P 817f6cdc7e2441d580c08e62681c6160: Deleting consensus metadata
I20260928 02:35:19.047629 10605 catalog_manager.cc:4841] TS 817f6cdc7e2441d580c08e62681c6160 (127.0.98.3:33855): tablet 43c45b848adc46b0806f8b3316bc7479 (table test-table [id=bad0831a189b4e4f99cb90fd7c6dde70]) successfully deleted
I20260928 02:35:19.047658 10607 catalog_manager.cc:4841] TS bc67fa0601014b97a2da1fc2adea0d42 (127.0.98.2:40709): tablet 43c45b848adc46b0806f8b3316bc7479 (table test-table [id=bad0831a189b4e4f99cb90fd7c6dde70]) successfully deleted
W20260928 02:35:19.049522 10607 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39257 (error 108)
I20260928 02:35:19.049738 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 10817
I20260928 02:35:19.050274 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:35:19.049802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39257 (local address 127.0.0.1:43400)
0928 02:35:19.050210 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0928 02:35:19.050244 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391}
W20260928 02:35:19.050573 10607 catalog_manager.cc:4584] TS 76dd3d866ea24db992ed9a62fc09866f (127.0.98.1:39257): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 43c45b848adc46b0806f8b3316bc7479: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39257: connect: Connection refused (error 111)
I20260928 02:35:19.053898 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 10951
I20260928 02:35:19.057617 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 10589
2026-09-28T02:35:19Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1921 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:35:19 dist-test-slave-fz47 krb5kdc[11124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:35:19 dist-test-slave-fz47 krb5kdc[11124](info): set up 2 sockets
Sep 28 02:35:19 dist-test-slave-fz47 krb5kdc[11124](info): commencing operation
krb5kdc: starting...
W20260928 02:35:20.001850 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:35:20 dist-test-slave-fz47 krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:35:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:35:20Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260928 02:35:20.147480 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskyt1071/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/postgres -l logfile start
2026-09-28 02:35:22.313 UTC [11154] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 02:35:22.313 UTC [11154] LOG: listening on IPv4 address "127.0.98.20", port 53975
2026-09-28 02:35:22.316 UTC [11154] LOG: listening on Unix socket "/tmp/.s.PGSQL.53975"
2026-09-28 02:35:22.341 UTC [11157] LOG: database system was shut down at 2026-09-28 02:35:21 UTC
2026-09-28 02:35:22.345 UTC [11154] LOG: database system is ready to accept connections
I20260928 02:35:23.205834 392 mini_postgres.cc:96] Postgres bound to 53975
2026-09-28 02:35:23.211 UTC [11165] FATAL: database "slave" does not exist
127.0.98.20:53975 - accepting connections
I20260928 02:35:23.211902 392 mini_ranger.cc:161] Starting Ranger...
I20260928 02:35:23.225286 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 02:35:23.643016 392 mini_ranger.cc:87] Created ranger Postgres database
I20260928 02:35:23.643085 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ranger-admin
2026-09-28 02:35:23,907 [I] DB FLAVOR :POSTGRES
2026-09-28 02:35:23,907 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:35:23,907 [I] Checking connection..
2026-09-28 02:35:23,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 02:35:24,144 [I] Checking connection passed.
2026-09-28 02:35:24,145 [I] --------- Verifying version history table ---------
2026-09-28 02:35:24,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:35:24,363 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:35:24,363 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:35:24,606 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:35:24,606 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 02:35:24,607 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53975/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 02:35:24.849 UTC [11229] WARNING: there is no transaction in progress
2026-09-28 02:35:24,854 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 02:35:24,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:35:25,101 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 02:35:25,101 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:35:25,101 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 02:35:25,344 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 02:35:25,588 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-fz47.c.gcp-upstream.internal','N') ;"
2026-09-28 02:35:25,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:35:26,075 [I] Table x_portal_user does not exist in database ranger
2026-09-28 02:35:26,075 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:35:26,328 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 02:35:26,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:35:26,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:35:26,800 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 02:35:26,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53975/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 02:35:27.693 UTC [11337] WARNING: there is no transaction in progress
2026-09-28 02:35:27.707 UTC [11337] WARNING: there is no transaction in progress
2026-09-28 02:35:27.722 UTC [11337] WARNING: there is no transaction in progress
2026-09-28 02:35:27.736 UTC [11337] WARNING: there is no transaction in progress
2026-09-28 02:35:27.751 UTC [11337] WARNING: there is no transaction in progress
2026-09-28 02:35:27.767 UTC [11337] WARNING: there is no transaction in progress
2026-09-28 02:35:27.857 UTC [11337] WARNING: there is no transaction in progress
2026-09-28 02:35:27.868 UTC [11337] WARNING: there is no transaction in progress
2026-09-28 02:35:27.882 UTC [11337] WARNING: there is no transaction in progress
2026-09-28 02:35:27.892 UTC [11337] WARNING: there is no transaction in progress
2026-09-28 02:35:27.902 UTC [11337] WARNING: there is no transaction in progress
2026-09-28 02:35:27.916 UTC [11337] WARNING: there is no transaction in progress
2026-09-28 02:35:27.925 UTC [11337] WARNING: there is no transaction in progress
2026-09-28 02:35:27.931 UTC [11337] WARNING: there is no transaction in progress
2026-09-28 02:35:27.937 UTC [11337] WARNING: there is no transaction in progress
2026-09-28 02:35:27.943 UTC [11337] WARNING: there is no transaction in progress
2026-09-28 02:35:28,525 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 02:35:28,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 02:35:28,765 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 02:35:28,765 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:35:29,018 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 02:35:29,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:35:29,275 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 02:35:29,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:35:29,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:35:29,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-fz47.c.gcp-upstream.internal';"
2026-09-28 02:35:29,989 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 02:35:29,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53975/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 02:35:30,232 [I] DB_PATCHES have already been applied
I20260928 02:35:30.237928 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskyt1071/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 02:35:30.238015 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260928 02:35:30.240783 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:42403
Sep 28, 2026 2:35:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:35:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:35:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:35:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:35:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:35:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:35:31 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 02:35:31.232975 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.021s user 0.000s sys 0.008s
Sep 28, 2026 2:35:50 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:35:50 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 28, 2026 2:35:50 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:35:50 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:35:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:35:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:35:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:35:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:35:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:35:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:35:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:35:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:35:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:35:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:35:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:35:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:35:50 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:35:51 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 02:35:52.288204 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 02:35:52 dist-test-slave-fz47 krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562952, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 02:35:52.376646 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 02:35:52.384430 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 02:35:53.156077 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 02:35:53.156157 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ranger-kms
I20260928 02:35:53.156172 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:53975
2026-09-28 02:35:53,309 [I] DB FLAVOR :POSTGRES
2026-09-28 02:35:53,309 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:35:53,309 [I] Checking connection
2026-09-28 02:35:53,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53975/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 02:35:53,553 [I] connection success
2026-09-28 02:35:53,553 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 02:35:53,553 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 02:35:53,553 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53975/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 02:35:53,778 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 02:35:53,778 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:35:53,778 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 02:35:53,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53975/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 02:35:54,027 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 02:35:54.504964 392 mini_ranger_kms.cc:310] Created kms service
I20260928 02:35:54.683349 392 mini_ranger_kms.cc:326] Created kudu user
I20260928 02:35:54.723961 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 02:35:54 dist-test-slave-fz47 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562952, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 02:35:55.081117 392 mini_ranger_kms.cc:384] Added ranger policy
I20260928 02:35:55.081228 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 02:35:55.081246 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260928 02:35:55.084527 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11517
I20260928 02:35:55.084651 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:39679
Sep 28, 2026 2:35:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:35:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:35:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:35:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:35:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:35:56 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 02:35:57.031497 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.655s user 0.000s sys 0.136s
I20260928 02:35:57.031589 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 02:35:57.031665 392 mini_ranger_kms.cc:401] 127.0.98.20:39679/kms/v1/keys
Sep 28 02:35:57 dist-test-slave-fz47 krb5kdc[11124](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790562957, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 02:35:57 dist-test-slave-fz47 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562957, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 02:35:58 dist-test-slave-fz47 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562957, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 02:36:00 dist-test-slave-fz47 krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 02:36:00.156168 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46425
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46425
--ranger_config_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39679/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:36:00.235607 11568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:36:00.235841 11568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:36:00.235864 11568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:36:00.237190 11568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:36:00.237244 11568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:36:00.237285 11568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:36:00.237303 11568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:36:00.237318 11568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:36:00.238952 11568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:39679/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46425
--ranger_config_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:36:00.239159 11568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:36:00.239343 11568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:36:00.244547 11574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:00.244560 11573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:00.244632 11576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:00.244684 11568 server_base.cc:1034] running on GCE node
I20260928 02:36:00.245026 11568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:36:00.245359 11568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:36:00.246587 11568 hybrid_clock.cc:648] HybridClock initialized: now 1790562960246567 us; error 40 us; skew 500 ppm
Sep 28 02:36:00 dist-test-slave-fz47 krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:36:00.252681 11568 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:36:00.253232 11568 webserver.cc:460] Webserver started at http://127.0.98.62:33611/ using document root <none> and password file <none>
I20260928 02:36:00.253515 11568 fs_manager.cc:359] Metadata directory not provided
I20260928 02:36:00.253561 11568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:36:00.253688 11568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:36:00 dist-test-slave-fz47 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 02:36:00.302271 11568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "6b9d4ac93354452caa8f4a12568b589b"
format_stamp: "Formatted at 2026-09-28 02:36:00 on dist-test-slave-fz47"
server_key: "ee141bf75f69d5421dc37e0954ca5955"
server_key_iv: "dad9c363dc32c4de94102cc197fe3583"
server_key_version: "kuduclusterkey@0"
I20260928 02:36:00.302735 11568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "6b9d4ac93354452caa8f4a12568b589b"
format_stamp: "Formatted at 2026-09-28 02:36:00 on dist-test-slave-fz47"
server_key: "ee141bf75f69d5421dc37e0954ca5955"
server_key_iv: "dad9c363dc32c4de94102cc197fe3583"
server_key_version: "kuduclusterkey@0"
I20260928 02:36:00.304787 11568 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:36:00.320079 11583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:00.322707 11568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:36:00.322825 11568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "6b9d4ac93354452caa8f4a12568b589b"
format_stamp: "Formatted at 2026-09-28 02:36:00 on dist-test-slave-fz47"
server_key: "ee141bf75f69d5421dc37e0954ca5955"
server_key_iv: "dad9c363dc32c4de94102cc197fe3583"
server_key_version: "kuduclusterkey@0"
I20260928 02:36:00.322924 11568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:36:00.370146 11568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:36:00.371974 11568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:36:00.372110 11568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:36:00.376797 11568 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46425
I20260928 02:36:00.376819 11635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46425 every 8 connection(s)
I20260928 02:36:00.377197 11568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:36:00.377828 11636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:36:00.379962 11636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b9d4ac93354452caa8f4a12568b589b: Bootstrap starting.
I20260928 02:36:00.380501 11636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b9d4ac93354452caa8f4a12568b589b: Neither blocks nor log segments found. Creating new log.
I20260928 02:36:00.380729 11636 log.cc:824] T 00000000000000000000000000000000 P 6b9d4ac93354452caa8f4a12568b589b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:36:00.381341 11636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b9d4ac93354452caa8f4a12568b589b: No bootstrap required, opened a new log
I20260928 02:36:00.381712 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 11568
I20260928 02:36:00.381824 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:36:00.382000 392 external_mini_cluster.cc:1428] Setting key c43e31dd7543ff6837e954237ee0737f
I20260928 02:36:00.383639 11636 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b9d4ac93354452caa8f4a12568b589b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b9d4ac93354452caa8f4a12568b589b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46425 } }
I20260928 02:36:00.383770 11636 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b9d4ac93354452caa8f4a12568b589b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:36:00.383801 11636 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b9d4ac93354452caa8f4a12568b589b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9d4ac93354452caa8f4a12568b589b, State: Initialized, Role: FOLLOWER
I20260928 02:36:00.383919 11636 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b9d4ac93354452caa8f4a12568b589b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b9d4ac93354452caa8f4a12568b589b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46425 } }
I20260928 02:36:00.383996 11636 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b9d4ac93354452caa8f4a12568b589b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:36:00.384029 11636 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b9d4ac93354452caa8f4a12568b589b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:36:00.384073 11636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b9d4ac93354452caa8f4a12568b589b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:36:00.384529 11636 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b9d4ac93354452caa8f4a12568b589b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b9d4ac93354452caa8f4a12568b589b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46425 } }
I20260928 02:36:00.384615 11636 leader_election.cc:302] T 00000000000000000000000000000000 P 6b9d4ac93354452caa8f4a12568b589b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b9d4ac93354452caa8f4a12568b589b; no voters:
I20260928 02:36:00.384757 11636 leader_election.cc:288] T 00000000000000000000000000000000 P 6b9d4ac93354452caa8f4a12568b589b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:36:00.384809 11641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b9d4ac93354452caa8f4a12568b589b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:36:00.384922 11641 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b9d4ac93354452caa8f4a12568b589b [term 1 LEADER]: Becoming Leader. State: Replica: 6b9d4ac93354452caa8f4a12568b589b, State: Running, Role: LEADER
I20260928 02:36:00.384996 11636 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b9d4ac93354452caa8f4a12568b589b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:36:00.385007 11641 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b9d4ac93354452caa8f4a12568b589b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b9d4ac93354452caa8f4a12568b589b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46425 } }
I20260928 02:36:00.387403 11643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b9d4ac93354452caa8f4a12568b589b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b9d4ac93354452caa8f4a12568b589b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b9d4ac93354452caa8f4a12568b589b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46425 } } }
I20260928 02:36:00.387488 11643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b9d4ac93354452caa8f4a12568b589b [sys.catalog]: This master's current role is: LEADER
I20260928 02:36:00.387615 11636 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 02:36:00.390331 11642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b9d4ac93354452caa8f4a12568b589b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b9d4ac93354452caa8f4a12568b589b. Latest consensus state: current_term: 1 leader_uuid: "6b9d4ac93354452caa8f4a12568b589b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b9d4ac93354452caa8f4a12568b589b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46425 } } }
I20260928 02:36:00.390558 11642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b9d4ac93354452caa8f4a12568b589b [sys.catalog]: This master's current role is: LEADER
I20260928 02:36:00.390808 11655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:36:00.391481 11655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 02:36:00 dist-test-slave-fz47 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:36:00.394881 11655 catalog_manager.cc:1347] Generated new cluster ID: b67b8079024740cc9df233e638b818d0
I20260928 02:36:00.394932 11655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:36:00.399299 11640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:00.382478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54900 (local address 127.0.98.62:46425)
0928 02:36:00.385068 (+ 2590us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:00.385077 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:00.385108 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:36:00.385223 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:00.385229 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:00.385302 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:00.385439 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:00.386136 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:00.390483 (+ 4347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:00.391693 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:00.392484 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:00.392616 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:00.395417 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:00.395421 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:00.395422 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:00.395432 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:00.397316 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:00.397727 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:00.397730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:00.397731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:00.397824 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:00.398014 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:00.398015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:00.398016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:00.398335 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:00.398461 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:00.398954 (+ 493us) server_negotiation.cc:299] Negotiation successful
0928 02:36:00.399079 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2547,"thread_start_us":57,"threads_started":1}
I20260928 02:36:00.401341 11655 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:36:00.402004 11655 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:36:00.409324 11655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b9d4ac93354452caa8f4a12568b589b: Generated new TSK 0
I20260928 02:36:00.409584 11655 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 02:36:01.108176 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39679/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46425
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:36:01.192515 11681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:36:01.192718 11681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:36:01.192746 11681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:36:01.193928 11681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:36:01.193972 11681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:36:01.194007 11681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:36:01.195662 11681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:39679/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:36:01.195883 11681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:36:01.196106 11681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:36:01.200836 11686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:01.200846 11687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:01.200975 11681 server_base.cc:1034] running on GCE node
W20260928 02:36:01.201156 11689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:01.201362 11681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:36:01.201622 11681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:36:01.202790 11681 hybrid_clock.cc:648] HybridClock initialized: now 1790562961202772 us; error 31 us; skew 500 ppm
Sep 28 02:36:01 dist-test-slave-fz47 krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:36:01.205703 11681 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 02:36:01.206230 11681 webserver.cc:460] Webserver started at http://127.0.98.1:41139/ using document root <none> and password file <none>
I20260928 02:36:01.206482 11681 fs_manager.cc:359] Metadata directory not provided
I20260928 02:36:01.206552 11681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:36:01.206684 11681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:36:01 dist-test-slave-fz47 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 02:36:01.220469 11681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-0/data/instance:
uuid: "b08aacc5f2fc464d91e13586f7257802"
format_stamp: "Formatted at 2026-09-28 02:36:01 on dist-test-slave-fz47"
server_key: "3c369860eb44c3e9758b2c2d1365717b"
server_key_iv: "defcae0793cf2bfebb9299f45cd6f69d"
server_key_version: "kuduclusterkey@0"
I20260928 02:36:01.220893 11681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b08aacc5f2fc464d91e13586f7257802"
format_stamp: "Formatted at 2026-09-28 02:36:01 on dist-test-slave-fz47"
server_key: "3c369860eb44c3e9758b2c2d1365717b"
server_key_iv: "defcae0793cf2bfebb9299f45cd6f69d"
server_key_version: "kuduclusterkey@0"
I20260928 02:36:01.222608 11681 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 02:36:01.237406 11696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:01.237666 11681 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260928 02:36:01.237778 11681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "b08aacc5f2fc464d91e13586f7257802"
format_stamp: "Formatted at 2026-09-28 02:36:01 on dist-test-slave-fz47"
server_key: "3c369860eb44c3e9758b2c2d1365717b"
server_key_iv: "defcae0793cf2bfebb9299f45cd6f69d"
server_key_version: "kuduclusterkey@0"
I20260928 02:36:01.237880 11681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:36:01.260762 11681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:36:01.263100 11681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:36:01.263293 11681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:36:01.263571 11681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:36:01.263980 11681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:36:01.264029 11681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:01.264060 11681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:36:01.264109 11681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:01.271725 11681 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33623
I20260928 02:36:01.272219 11681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
I20260928 02:36:01.273823 11811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33623 every 8 connection(s)
I20260928 02:36:01.274596 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 11681
I20260928 02:36:01.274701 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-0/wal/instance
I20260928 02:36:01.274881 392 external_mini_cluster.cc:1428] Setting key 161cb24ac16ee9c35fa10607394f5b51
Sep 28 02:36:01 dist-test-slave-fz47 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:36:01.287850 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:01.274568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46425 (local address 127.0.98.1:34601)
0928 02:36:01.277438 (+ 2870us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:01.277494 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:01.278063 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:01.278408 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:01.278413 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:01.278660 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:36:01.278935 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:01.278941 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.279948 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.279950 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:01.281221 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:01.281223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.281906 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.281907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:01.282070 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:01.282652 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:36:01.282662 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:36:01.284283 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:36:01.286232 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:01.286238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:01.286240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:01.286541 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:01.286744 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:01.286745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:01.286745 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:01.286875 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:01.287157 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:36:01.287163 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:36:01.287341 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 02:36:01.287447 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 02:36:01.287614 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2826,"thread_start_us":84,"threads_started":1}
I20260928 02:36:01.288717 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:01.274574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34601 (local address 127.0.98.62:46425)
0928 02:36:01.277369 (+ 2795us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:01.277372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:01.278191 (+ 819us) server_negotiation.cc:407] Connection header received
0928 02:36:01.278217 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:01.278218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:01.278231 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:01.278288 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:01.279040 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.279845 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.281466 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.281781 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.282317 (+ 536us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:01.284381 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:01.284383 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:01.284383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:01.284391 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:01.286146 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:01.286621 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:01.286622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:01.286623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:01.286681 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:01.286941 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:01.286941 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:01.286942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:01.287074 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:01.287131 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:01.288539 (+ 1408us) server_negotiation.cc:299] Negotiation successful
0928 02:36:01.288621 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2765,"thread_start_us":80,"threads_started":1}
I20260928 02:36:01.289279 11812 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46425
I20260928 02:36:01.289373 11812 heartbeater.cc:464] Registering TS with master...
I20260928 02:36:01.289568 11812 heartbeater.cc:511] Master 127.0.98.62:46425 requested a full tablet report, sending...
I20260928 02:36:01.290143 11600 ts_manager.cc:194] Registered new tserver with Master: b08aacc5f2fc464d91e13586f7257802 (127.0.98.1:33623)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 02:36:01.291031 11600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34601
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 02:36:01.316174 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39679/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46425
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:36:01.424840 11818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:36:01.425045 11818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:36:01.425074 11818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:36:01.426208 11818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:36:01.426249 11818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:36:01.426286 11818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:36:01.427846 11818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:39679/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:36:01.428063 11818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:36:01.428292 11818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:36:01.432934 11828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:01.432960 11826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:01.433372 11825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:01.433605 11818 server_base.cc:1034] running on GCE node
I20260928 02:36:01.433795 11818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:36:01.434003 11818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:36:01.441311 11818 hybrid_clock.cc:648] HybridClock initialized: now 1790562961441288 us; error 29 us; skew 500 ppm
Sep 28 02:36:01 dist-test-slave-fz47 krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:36:01.443821 11818 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 02:36:01.444278 11818 webserver.cc:460] Webserver started at http://127.0.98.2:42069/ using document root <none> and password file <none>
I20260928 02:36:01.444479 11818 fs_manager.cc:359] Metadata directory not provided
I20260928 02:36:01.444525 11818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:36:01.444653 11818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:36:01 dist-test-slave-fz47 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 02:36:01 dist-test-slave-fz47 krb5kdc[11124](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790562961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:36:01.459787 11818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-1/data/instance:
uuid: "48e42310c9854ac480b1abeb10b59b77"
format_stamp: "Formatted at 2026-09-28 02:36:01 on dist-test-slave-fz47"
server_key: "2c18c17046acc444ec55c3da2d45f06c"
server_key_iv: "351597a6372bf4ce0cfc6d26e914d80a"
server_key_version: "kuduclusterkey@0"
I20260928 02:36:01.460243 11818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-1/wal/instance:
uuid: "48e42310c9854ac480b1abeb10b59b77"
format_stamp: "Formatted at 2026-09-28 02:36:01 on dist-test-slave-fz47"
server_key: "2c18c17046acc444ec55c3da2d45f06c"
server_key_iv: "351597a6372bf4ce0cfc6d26e914d80a"
server_key_version: "kuduclusterkey@0"
I20260928 02:36:01.464368 11818 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:36:01.482662 11835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:01.482858 11818 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260928 02:36:01.482955 11818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "48e42310c9854ac480b1abeb10b59b77"
format_stamp: "Formatted at 2026-09-28 02:36:01 on dist-test-slave-fz47"
server_key: "2c18c17046acc444ec55c3da2d45f06c"
server_key_iv: "351597a6372bf4ce0cfc6d26e914d80a"
server_key_version: "kuduclusterkey@0"
I20260928 02:36:01.483060 11818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:36:01.514550 11818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:36:01.516979 11818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:36:01.517128 11818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:36:01.517375 11818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:36:01.517743 11818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:36:01.517795 11818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:01.517830 11818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:36:01.517846 11818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:01.525329 11818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38223
I20260928 02:36:01.525748 11818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
I20260928 02:36:01.528683 11948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38223 every 8 connection(s)
Sep 28 02:36:01 dist-test-slave-fz47 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:36:01.534235 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 11818
I20260928 02:36:01.534332 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-1/wal/instance
I20260928 02:36:01.534524 392 external_mini_cluster.cc:1428] Setting key 0632eb5a6c86ee6ec67fe9f0076fda46
I20260928 02:36:01.537635 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:01.529281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59501 (local address 127.0.98.62:46425)
0928 02:36:01.529327 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:01.529329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:01.529539 (+ 210us) server_negotiation.cc:407] Connection header received
0928 02:36:01.530032 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:01.530033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:01.530045 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:01.530098 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:01.530687 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.531195 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.532224 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.532417 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.532470 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:01.534460 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:01.534461 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:01.534462 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:01.534469 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:01.535977 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:01.536417 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:01.536418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:01.536418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:01.536472 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:01.536729 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:01.536730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:01.536730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:01.536896 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:01.536949 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:01.537446 (+ 497us) server_negotiation.cc:299] Negotiation successful
0928 02:36:01.537540 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:36:01.538125 11951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:01.529210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46425 (local address 127.0.98.2:59501)
0928 02:36:01.529414 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:01.529454 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:01.529937 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:01.530173 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:01.530177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:01.530400 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:36:01.530596 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:01.530600 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.531279 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.531280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:01.532140 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:01.532143 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.532482 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.532483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:01.532596 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:01.533076 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:36:01.533084 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:36:01.534352 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:36:01.536067 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:01.536074 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:01.536076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:01.536336 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:01.536533 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:01.536533 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:01.536534 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:01.536657 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:01.537011 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:36:01.537020 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:36:01.537294 (+ 274us) client_negotiation.cc:769] Sending connection context
0928 02:36:01.537724 (+ 430us) client_negotiation.cc:240] Negotiation successful
0928 02:36:01.537897 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":105,"threads_started":1}
I20260928 02:36:01.538630 11949 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46425
I20260928 02:36:01.538721 11949 heartbeater.cc:464] Registering TS with master...
I20260928 02:36:01.538887 11949 heartbeater.cc:511] Master 127.0.98.62:46425 requested a full tablet report, sending...
I20260928 02:36:01.539335 11600 ts_manager.cc:194] Registered new tserver with Master: 48e42310c9854ac480b1abeb10b59b77 (127.0.98.2:38223)
I20260928 02:36:01.539675 11600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59501
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 02:36:01.572479 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:39679/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46425
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:36:01.677130 11955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:36:01.677479 11955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:36:01.677560 11955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:36:01.679217 11955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:36:01.679277 11955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:36:01.679314 11955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:36:01.681437 11955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:39679/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:36:01.681669 11955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:36:01.681913 11955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:36:01.687999 11960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:01.687995 11961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:01.688282 11963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:01.688489 11955 server_base.cc:1034] running on GCE node
I20260928 02:36:01.688705 11955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:36:01.688978 11955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:36:01.690155 11955 hybrid_clock.cc:648] HybridClock initialized: now 1790562961690115 us; error 50 us; skew 500 ppm
Sep 28 02:36:01 dist-test-slave-fz47 krb5kdc[11124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:36:01.693320 11955 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 02:36:01.693795 11955 webserver.cc:460] Webserver started at http://127.0.98.3:44465/ using document root <none> and password file <none>
I20260928 02:36:01.694056 11955 fs_manager.cc:359] Metadata directory not provided
I20260928 02:36:01.694110 11955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:36:01.694231 11955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:36:01 dist-test-slave-fz47 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 02:36:01.715432 11955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-2/data/instance:
uuid: "a26973bfe71a4fdca78e6a47abf8601f"
format_stamp: "Formatted at 2026-09-28 02:36:01 on dist-test-slave-fz47"
server_key: "816b0e3d6b86adb2493c12f886cbaacd"
server_key_iv: "4abed204408766c4ca0e78b22a5fde22"
server_key_version: "kuduclusterkey@0"
I20260928 02:36:01.715874 11955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a26973bfe71a4fdca78e6a47abf8601f"
format_stamp: "Formatted at 2026-09-28 02:36:01 on dist-test-slave-fz47"
server_key: "816b0e3d6b86adb2493c12f886cbaacd"
server_key_iv: "4abed204408766c4ca0e78b22a5fde22"
server_key_version: "kuduclusterkey@0"
I20260928 02:36:01.725704 11955 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.004s sys 0.000s
I20260928 02:36:01.740011 11970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:01.740191 11955 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 02:36:01.740283 11955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "a26973bfe71a4fdca78e6a47abf8601f"
format_stamp: "Formatted at 2026-09-28 02:36:01 on dist-test-slave-fz47"
server_key: "816b0e3d6b86adb2493c12f886cbaacd"
server_key_iv: "4abed204408766c4ca0e78b22a5fde22"
server_key_version: "kuduclusterkey@0"
I20260928 02:36:01.740386 11955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:36:01.759193 11955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:36:01.761618 11955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:36:01.761755 11955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:36:01.761988 11955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:36:01.763463 11955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:36:01.763511 11955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:01.763545 11955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:36:01.763566 11955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:01.770519 11955 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36399
I20260928 02:36:01.770557 12083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36399 every 8 connection(s)
I20260928 02:36:01.770910 11955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 02:36:01 dist-test-slave-fz47 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:36:01.779238 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 11955
I20260928 02:36:01.779348 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-2/wal/instance
I20260928 02:36:01.779526 392 external_mini_cluster.cc:1428] Setting key ab41241741ac8798631638d2ace180e7
I20260928 02:36:01.780097 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:01.771717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60189 (local address 127.0.98.62:46425)
0928 02:36:01.771775 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:01.771777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:01.771964 (+ 187us) server_negotiation.cc:407] Connection header received
0928 02:36:01.772444 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:01.772445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:01.772456 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:01.772532 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:01.773049 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.773808 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.774958 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.775232 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.775301 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:01.777311 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:01.777313 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:01.777313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:01.777321 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:01.778756 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:01.779132 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:01.779134 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:01.779134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:01.779186 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:01.779445 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:01.779446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:01.779446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:01.779557 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:01.779613 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:01.779926 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 02:36:01.780005 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 02:36:01.780210 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:01.771672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46425 (local address 127.0.98.3:60189)
0928 02:36:01.771842 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:01.771882 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:01.772343 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:01.772586 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:01.772589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:01.772765 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:36:01.772959 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:01.772963 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.773889 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.773890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:01.774885 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:01.774887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.775319 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.775320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:01.775449 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:01.775897 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:36:01.775906 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:36:01.777186 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:36:01.778833 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:01.778838 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:01.778840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:01.779044 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:01.779239 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:01.779239 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:01.779239 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:01.779355 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:01.779611 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:36:01.779618 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:36:01.779786 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 02:36:01.779887 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 02:36:01.780031 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
I20260928 02:36:01.780637 12084 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46425
I20260928 02:36:01.780730 12084 heartbeater.cc:464] Registering TS with master...
I20260928 02:36:01.780902 12084 heartbeater.cc:511] Master 127.0.98.62:46425 requested a full tablet report, sending...
I20260928 02:36:01.781244 11600 ts_manager.cc:194] Registered new tserver with Master: a26973bfe71a4fdca78e6a47abf8601f (127.0.98.3:36399)
I20260928 02:36:01.781774 11600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60189
I20260928 02:36:01.798389 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:36:01.807749 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:01.800025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38060 (local address 127.0.98.62:46425)
0928 02:36:01.800069 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:01.800071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:01.800200 (+ 129us) server_negotiation.cc:407] Connection header received
0928 02:36:01.800255 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:01.800256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:01.800267 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:01.800331 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:01.801029 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.801861 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.802914 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.803203 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.803243 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:01.804819 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:01.804820 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:01.804821 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:01.804828 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:01.806492 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:01.806872 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:01.806873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:01.806874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:01.806927 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:01.807128 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:01.807129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:01.807129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:01.807275 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:01.807351 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:01.807575 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 02:36:01.807650 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:36:01.809851 11600 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:36:01.812287 11600 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:36:01.823244 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:01.816979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36399 (local address 127.0.0.1:49044)
0928 02:36:01.818281 (+ 1302us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:01.818321 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:01.818401 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:01.818687 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:01.818690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:01.818692 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:36:01.818764 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:01.818766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.820318 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.820319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:01.821638 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:01.821641 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.822977 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.822978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:01.823058 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:01.823070 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 02:36:01.823125 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 02:36:01.823167 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1274,"thread_start_us":91,"threads_started":1}
I20260928 02:36:01.823673 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:01.817595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49044 (local address 127.0.98.3:36399)
0928 02:36:01.818156 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:01.818161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:01.818481 (+ 320us) server_negotiation.cc:407] Connection header received
0928 02:36:01.818523 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:01.818525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:01.818541 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:01.818609 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:36:01.818834 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.820238 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.821785 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.822810 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.822875 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:01.822946 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:01.823554 (+ 608us) server_negotiation.cc:299] Negotiation successful
0928 02:36:01.823594 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":80,"threads_started":1}
I20260928 02:36:01.824420 12096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:01.816980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38223 (local address 127.0.0.1:38708)
0928 02:36:01.820666 (+ 3686us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:01.820675 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:01.820728 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:01.820995 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:01.820995 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:01.820997 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:36:01.821030 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:01.821031 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.822194 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.822195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:01.823222 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:01.823224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.824208 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.824208 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:01.824267 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:01.824271 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:36:01.824314 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 02:36:01.824349 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3671,"thread_start_us":132,"threads_started":1}
I20260928 02:36:01.824704 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:01.817298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38708 (local address 127.0.98.2:38223)
0928 02:36:01.820601 (+ 3303us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:01.820605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:01.820774 (+ 169us) server_negotiation.cc:407] Connection header received
0928 02:36:01.820813 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:01.820815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:01.820830 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:01.820922 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:36:01.821092 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.822116 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.823321 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.824056 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.824108 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:01.824178 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:01.824593 (+ 415us) server_negotiation.cc:299] Negotiation successful
0928 02:36:01.824631 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3269,"thread_start_us":83,"threads_started":1}
I20260928 02:36:01.824846 12018 tablet_service.cc:1467] Processing CreateTablet for tablet d72a166640af433fb6519fa32c40f6a2 (DEFAULT_TABLE table=test-table [id=76370d684c2c47729fee7314d831a161]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:36:01.825189 12018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d72a166640af433fb6519fa32c40f6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:36:01.825289 11883 tablet_service.cc:1467] Processing CreateTablet for tablet d72a166640af433fb6519fa32c40f6a2 (DEFAULT_TABLE table=test-table [id=76370d684c2c47729fee7314d831a161]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:36:01.825572 11883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d72a166640af433fb6519fa32c40f6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:36:01.829564 12103 tablet_bootstrap.cc:492] T d72a166640af433fb6519fa32c40f6a2 P 48e42310c9854ac480b1abeb10b59b77: Bootstrap starting.
I20260928 02:36:01.830487 12103 tablet_bootstrap.cc:654] T d72a166640af433fb6519fa32c40f6a2 P 48e42310c9854ac480b1abeb10b59b77: Neither blocks nor log segments found. Creating new log.
I20260928 02:36:01.830778 12103 log.cc:824] T d72a166640af433fb6519fa32c40f6a2 P 48e42310c9854ac480b1abeb10b59b77: Log is configured to *not* fsync() on all Append() calls
I20260928 02:36:01.831445 12103 tablet_bootstrap.cc:492] T d72a166640af433fb6519fa32c40f6a2 P 48e42310c9854ac480b1abeb10b59b77: No bootstrap required, opened a new log
I20260928 02:36:01.831512 12103 ts_tablet_manager.cc:1397] T d72a166640af433fb6519fa32c40f6a2 P 48e42310c9854ac480b1abeb10b59b77: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:36:01.833348 12103 raft_consensus.cc:348] T d72a166640af433fb6519fa32c40f6a2 P 48e42310c9854ac480b1abeb10b59b77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26973bfe71a4fdca78e6a47abf8601f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36399 } } peers { permanent_uuid: "48e42310c9854ac480b1abeb10b59b77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38223 } } peers { permanent_uuid: "b08aacc5f2fc464d91e13586f7257802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33623 } }
I20260928 02:36:01.833519 12103 raft_consensus.cc:374] T d72a166640af433fb6519fa32c40f6a2 P 48e42310c9854ac480b1abeb10b59b77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:36:01.833559 12103 raft_consensus.cc:729] T d72a166640af433fb6519fa32c40f6a2 P 48e42310c9854ac480b1abeb10b59b77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e42310c9854ac480b1abeb10b59b77, State: Initialized, Role: FOLLOWER
I20260928 02:36:01.833663 12103 consensus_queue.cc:260] T d72a166640af433fb6519fa32c40f6a2 P 48e42310c9854ac480b1abeb10b59b77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26973bfe71a4fdca78e6a47abf8601f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36399 } } peers { permanent_uuid: "48e42310c9854ac480b1abeb10b59b77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38223 } } peers { permanent_uuid: "b08aacc5f2fc464d91e13586f7257802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33623 } }
I20260928 02:36:01.833904 12103 ts_tablet_manager.cc:1428] T d72a166640af433fb6519fa32c40f6a2 P 48e42310c9854ac480b1abeb10b59b77: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:36:01.834044 11949 heartbeater.cc:503] Master 127.0.98.62:46425 was elected leader, sending a full tablet report...
I20260928 02:36:01.834345 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:01.817231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33623 (local address 127.0.0.1:35764)
0928 02:36:01.823948 (+ 6717us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:01.823957 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:01.823989 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:01.829987 (+ 5998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:01.829988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:01.829991 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:36:01.830035 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:01.830037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.831414 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.831415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:01.832587 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:01.832589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.834119 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.834120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:01.834178 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:01.834182 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:36:01.834235 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 02:36:01.834274 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6706,"thread_start_us":104,"threads_started":1}
I20260928 02:36:01.834530 12102 tablet_bootstrap.cc:492] T d72a166640af433fb6519fa32c40f6a2 P a26973bfe71a4fdca78e6a47abf8601f: Bootstrap starting.
I20260928 02:36:01.835418 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:01.817755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35764 (local address 127.0.98.1:33623)
0928 02:36:01.818221 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:01.818224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:01.824042 (+ 5818us) server_negotiation.cc:407] Connection header received
0928 02:36:01.824079 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:01.824081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:01.824097 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:01.824160 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:36:01.830130 (+ 5970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.831309 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.832683 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.834011 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.834077 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:01.834176 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:01.834250 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 02:36:01.834286 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":48,"threads_started":1}
I20260928 02:36:01.835484 12102 tablet_bootstrap.cc:654] T d72a166640af433fb6519fa32c40f6a2 P a26973bfe71a4fdca78e6a47abf8601f: Neither blocks nor log segments found. Creating new log.
I20260928 02:36:01.835772 12102 log.cc:824] T d72a166640af433fb6519fa32c40f6a2 P a26973bfe71a4fdca78e6a47abf8601f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:36:01.836155 11746 tablet_service.cc:1467] Processing CreateTablet for tablet d72a166640af433fb6519fa32c40f6a2 (DEFAULT_TABLE table=test-table [id=76370d684c2c47729fee7314d831a161]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:36:01.836438 12102 tablet_bootstrap.cc:492] T d72a166640af433fb6519fa32c40f6a2 P a26973bfe71a4fdca78e6a47abf8601f: No bootstrap required, opened a new log
I20260928 02:36:01.836480 11746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d72a166640af433fb6519fa32c40f6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:36:01.836491 12102 ts_tablet_manager.cc:1397] T d72a166640af433fb6519fa32c40f6a2 P a26973bfe71a4fdca78e6a47abf8601f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:36:01.838406 12102 raft_consensus.cc:348] T d72a166640af433fb6519fa32c40f6a2 P a26973bfe71a4fdca78e6a47abf8601f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26973bfe71a4fdca78e6a47abf8601f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36399 } } peers { permanent_uuid: "48e42310c9854ac480b1abeb10b59b77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38223 } } peers { permanent_uuid: "b08aacc5f2fc464d91e13586f7257802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33623 } }
I20260928 02:36:01.838553 12102 raft_consensus.cc:374] T d72a166640af433fb6519fa32c40f6a2 P a26973bfe71a4fdca78e6a47abf8601f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:36:01.838585 12102 raft_consensus.cc:729] T d72a166640af433fb6519fa32c40f6a2 P a26973bfe71a4fdca78e6a47abf8601f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a26973bfe71a4fdca78e6a47abf8601f, State: Initialized, Role: FOLLOWER
I20260928 02:36:01.838687 12102 consensus_queue.cc:260] T d72a166640af433fb6519fa32c40f6a2 P a26973bfe71a4fdca78e6a47abf8601f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26973bfe71a4fdca78e6a47abf8601f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36399 } } peers { permanent_uuid: "48e42310c9854ac480b1abeb10b59b77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38223 } } peers { permanent_uuid: "b08aacc5f2fc464d91e13586f7257802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33623 } }
I20260928 02:36:01.838918 12102 ts_tablet_manager.cc:1428] T d72a166640af433fb6519fa32c40f6a2 P a26973bfe71a4fdca78e6a47abf8601f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:36:01.839030 12084 heartbeater.cc:503] Master 127.0.98.62:46425 was elected leader, sending a full tablet report...
I20260928 02:36:01.840118 12108 tablet_bootstrap.cc:492] T d72a166640af433fb6519fa32c40f6a2 P b08aacc5f2fc464d91e13586f7257802: Bootstrap starting.
I20260928 02:36:01.841061 12108 tablet_bootstrap.cc:654] T d72a166640af433fb6519fa32c40f6a2 P b08aacc5f2fc464d91e13586f7257802: Neither blocks nor log segments found. Creating new log.
I20260928 02:36:01.841327 12108 log.cc:824] T d72a166640af433fb6519fa32c40f6a2 P b08aacc5f2fc464d91e13586f7257802: Log is configured to *not* fsync() on all Append() calls
I20260928 02:36:01.841938 12108 tablet_bootstrap.cc:492] T d72a166640af433fb6519fa32c40f6a2 P b08aacc5f2fc464d91e13586f7257802: No bootstrap required, opened a new log
I20260928 02:36:01.841993 12108 ts_tablet_manager.cc:1397] T d72a166640af433fb6519fa32c40f6a2 P b08aacc5f2fc464d91e13586f7257802: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 02:36:01.843696 12108 raft_consensus.cc:348] T d72a166640af433fb6519fa32c40f6a2 P b08aacc5f2fc464d91e13586f7257802 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26973bfe71a4fdca78e6a47abf8601f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36399 } } peers { permanent_uuid: "48e42310c9854ac480b1abeb10b59b77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38223 } } peers { permanent_uuid: "b08aacc5f2fc464d91e13586f7257802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33623 } }
I20260928 02:36:01.843815 12108 raft_consensus.cc:374] T d72a166640af433fb6519fa32c40f6a2 P b08aacc5f2fc464d91e13586f7257802 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:36:01.843843 12108 raft_consensus.cc:729] T d72a166640af433fb6519fa32c40f6a2 P b08aacc5f2fc464d91e13586f7257802 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b08aacc5f2fc464d91e13586f7257802, State: Initialized, Role: FOLLOWER
I20260928 02:36:01.843961 12108 consensus_queue.cc:260] T d72a166640af433fb6519fa32c40f6a2 P b08aacc5f2fc464d91e13586f7257802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26973bfe71a4fdca78e6a47abf8601f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36399 } } peers { permanent_uuid: "48e42310c9854ac480b1abeb10b59b77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38223 } } peers { permanent_uuid: "b08aacc5f2fc464d91e13586f7257802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33623 } }
I20260928 02:36:01.844221 12108 ts_tablet_manager.cc:1428] T d72a166640af433fb6519fa32c40f6a2 P b08aacc5f2fc464d91e13586f7257802: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 02:36:01.844240 11812 heartbeater.cc:503] Master 127.0.98.62:46425 was elected leader, sending a full tablet report...
I20260928 02:36:01.941339 12110 raft_consensus.cc:482] T d72a166640af433fb6519fa32c40f6a2 P b08aacc5f2fc464d91e13586f7257802 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:36:01.941501 12110 raft_consensus.cc:504] T d72a166640af433fb6519fa32c40f6a2 P b08aacc5f2fc464d91e13586f7257802 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26973bfe71a4fdca78e6a47abf8601f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36399 } } peers { permanent_uuid: "48e42310c9854ac480b1abeb10b59b77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38223 } } peers { permanent_uuid: "b08aacc5f2fc464d91e13586f7257802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33623 } }
I20260928 02:36:01.941877 12110 leader_election.cc:288] T d72a166640af433fb6519fa32c40f6a2 P b08aacc5f2fc464d91e13586f7257802 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a26973bfe71a4fdca78e6a47abf8601f (127.0.98.3:36399), 48e42310c9854ac480b1abeb10b59b77 (127.0.98.2:38223)
I20260928 02:36:01.947475 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:01.942148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41417 (local address 127.0.98.2:38223)
0928 02:36:01.942192 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:01.942194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:01.943017 (+ 823us) server_negotiation.cc:407] Connection header received
0928 02:36:01.944131 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:01.944133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:01.944144 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:01.944172 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:36:01.944416 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.945110 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.946559 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.947108 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.947153 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:01.947163 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:01.947366 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 02:36:01.947405 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 02:36:01.947671 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:01.942085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38223 (local address 127.0.98.1:41417)
0928 02:36:01.942954 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:01.942964 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:01.944029 (+ 1065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:01.944240 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:01.944241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:01.944256 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:36:01.944334 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:01.944337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.945219 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.945220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:01.946455 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:01.946459 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.947219 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.947220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:01.947287 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:01.947291 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:36:01.947554 (+ 263us) client_negotiation.cc:240] Negotiation successful
0928 02:36:01.947600 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"thread_start_us":115,"threads_started":1}
I20260928 02:36:01.947942 11903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d72a166640af433fb6519fa32c40f6a2" candidate_uuid: "b08aacc5f2fc464d91e13586f7257802" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e42310c9854ac480b1abeb10b59b77" is_pre_election: true
I20260928 02:36:01.948107 11903 raft_consensus.cc:2454] T d72a166640af433fb6519fa32c40f6a2 P 48e42310c9854ac480b1abeb10b59b77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b08aacc5f2fc464d91e13586f7257802 in term 0.
I20260928 02:36:01.948333 11697 leader_election.cc:302] T d72a166640af433fb6519fa32c40f6a2 P b08aacc5f2fc464d91e13586f7257802 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e42310c9854ac480b1abeb10b59b77, b08aacc5f2fc464d91e13586f7257802; no voters:
I20260928 02:36:01.948474 12110 raft_consensus.cc:2781] T d72a166640af433fb6519fa32c40f6a2 P b08aacc5f2fc464d91e13586f7257802 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:36:01.948525 12110 raft_consensus.cc:482] T d72a166640af433fb6519fa32c40f6a2 P b08aacc5f2fc464d91e13586f7257802 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:36:01.948550 12110 raft_consensus.cc:3037] T d72a166640af433fb6519fa32c40f6a2 P b08aacc5f2fc464d91e13586f7257802 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:36:01.949880 12110 raft_consensus.cc:504] T d72a166640af433fb6519fa32c40f6a2 P b08aacc5f2fc464d91e13586f7257802 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26973bfe71a4fdca78e6a47abf8601f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36399 } } peers { permanent_uuid: "48e42310c9854ac480b1abeb10b59b77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38223 } } peers { permanent_uuid: "b08aacc5f2fc464d91e13586f7257802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33623 } }
I20260928 02:36:01.950019 12110 leader_election.cc:288] T d72a166640af433fb6519fa32c40f6a2 P b08aacc5f2fc464d91e13586f7257802 [CANDIDATE]: Term 1 election: Requested vote from peers a26973bfe71a4fdca78e6a47abf8601f (127.0.98.3:36399), 48e42310c9854ac480b1abeb10b59b77 (127.0.98.2:38223)
I20260928 02:36:01.950223 11903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d72a166640af433fb6519fa32c40f6a2" candidate_uuid: "b08aacc5f2fc464d91e13586f7257802" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e42310c9854ac480b1abeb10b59b77"
I20260928 02:36:01.950300 11903 raft_consensus.cc:3037] T d72a166640af433fb6519fa32c40f6a2 P 48e42310c9854ac480b1abeb10b59b77 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:36:01.950958 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:01.942682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48081 (local address 127.0.98.3:36399)
0928 02:36:01.942721 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:01.942722 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:01.947860 (+ 5138us) server_negotiation.cc:407] Connection header received
0928 02:36:01.947952 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:01.947953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:01.947964 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:01.947992 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:36:01.948137 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.948787 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.950067 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.950601 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.950646 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:01.950657 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:01.950853 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 02:36:01.950891 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:36:01.951054 11903 raft_consensus.cc:2454] T d72a166640af433fb6519fa32c40f6a2 P 48e42310c9854ac480b1abeb10b59b77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b08aacc5f2fc464d91e13586f7257802 in term 1.
I20260928 02:36:01.951172 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:01.942411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36399 (local address 127.0.98.1:48081)
0928 02:36:01.947816 (+ 5405us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:01.947825 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:01.947884 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:01.948047 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:01.948048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:01.948049 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:36:01.948075 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:01.948077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.948909 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.948910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:01.949972 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:01.949975 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.950707 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.950708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:01.950766 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:01.950769 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:36:01.951070 (+ 301us) client_negotiation.cc:240] Negotiation successful
0928 02:36:01.951109 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5384,"thread_start_us":69,"threads_started":1}
I20260928 02:36:01.951215 11697 leader_election.cc:302] T d72a166640af433fb6519fa32c40f6a2 P b08aacc5f2fc464d91e13586f7257802 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e42310c9854ac480b1abeb10b59b77, b08aacc5f2fc464d91e13586f7257802; no voters:
I20260928 02:36:01.951293 12110 raft_consensus.cc:2781] T d72a166640af433fb6519fa32c40f6a2 P b08aacc5f2fc464d91e13586f7257802 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:36:01.951423 12110 raft_consensus.cc:686] T d72a166640af433fb6519fa32c40f6a2 P b08aacc5f2fc464d91e13586f7257802 [term 1 LEADER]: Becoming Leader. State: Replica: b08aacc5f2fc464d91e13586f7257802, State: Running, Role: LEADER
I20260928 02:36:01.951515 12110 consensus_queue.cc:237] T d72a166640af433fb6519fa32c40f6a2 P b08aacc5f2fc464d91e13586f7257802 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26973bfe71a4fdca78e6a47abf8601f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36399 } } peers { permanent_uuid: "48e42310c9854ac480b1abeb10b59b77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38223 } } peers { permanent_uuid: "b08aacc5f2fc464d91e13586f7257802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33623 } }
I20260928 02:36:01.951534 12038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d72a166640af433fb6519fa32c40f6a2" candidate_uuid: "b08aacc5f2fc464d91e13586f7257802" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a26973bfe71a4fdca78e6a47abf8601f" is_pre_election: true
I20260928 02:36:01.951676 12038 raft_consensus.cc:2454] T d72a166640af433fb6519fa32c40f6a2 P a26973bfe71a4fdca78e6a47abf8601f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b08aacc5f2fc464d91e13586f7257802 in term 0.
I20260928 02:36:01.951814 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d72a166640af433fb6519fa32c40f6a2" candidate_uuid: "b08aacc5f2fc464d91e13586f7257802" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a26973bfe71a4fdca78e6a47abf8601f"
I20260928 02:36:01.951889 12037 raft_consensus.cc:3037] T d72a166640af433fb6519fa32c40f6a2 P a26973bfe71a4fdca78e6a47abf8601f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:36:01.952272 11600 catalog_manager.cc:5495] T d72a166640af433fb6519fa32c40f6a2 P b08aacc5f2fc464d91e13586f7257802 reported cstate change: term changed from 0 to 1, leader changed from <none> to b08aacc5f2fc464d91e13586f7257802 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b08aacc5f2fc464d91e13586f7257802" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a26973bfe71a4fdca78e6a47abf8601f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e42310c9854ac480b1abeb10b59b77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b08aacc5f2fc464d91e13586f7257802" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33623 } health_report { overall_health: HEALTHY } } }
I20260928 02:36:01.952696 12037 raft_consensus.cc:2454] T d72a166640af433fb6519fa32c40f6a2 P a26973bfe71a4fdca78e6a47abf8601f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b08aacc5f2fc464d91e13586f7257802 in term 1.
Sep 28 02:36:01 dist-test-slave-fz47 krb5kdc[11124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 02:36:01.979782 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:01.976645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35766 (local address 127.0.98.1:33623)
0928 02:36:01.976692 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:01.976694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:01.976701 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:36:01.976740 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:01.976741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:01.976764 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:01.976841 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:36:01.977019 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.977684 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.978714 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:01.978985 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:01.979040 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:01.979235 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:36:01.979369 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:36:01.979447 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:01.979681 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 02:36:01.979703 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:36:01.991133 11903 raft_consensus.cc:1261] T d72a166640af433fb6519fa32c40f6a2 P 48e42310c9854ac480b1abeb10b59b77 [term 1 FOLLOWER]: Refusing update from remote peer b08aacc5f2fc464d91e13586f7257802: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:36:01.991478 12037 raft_consensus.cc:1261] T d72a166640af433fb6519fa32c40f6a2 P a26973bfe71a4fdca78e6a47abf8601f [term 1 FOLLOWER]: Refusing update from remote peer b08aacc5f2fc464d91e13586f7257802: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:36:01.991775 12110 consensus_queue.cc:1035] T d72a166640af433fb6519fa32c40f6a2 P b08aacc5f2fc464d91e13586f7257802 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a26973bfe71a4fdca78e6a47abf8601f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:36:01.992129 12110 consensus_queue.cc:1035] T d72a166640af433fb6519fa32c40f6a2 P b08aacc5f2fc464d91e13586f7257802 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e42310c9854ac480b1abeb10b59b77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:36:02.000938 12116 mvcc.cc:205] Tried to move back new op lower bound from 7334145892311961600 to 7334145892153995264. Current Snapshot: MvccSnapshot[applied={T|T < 7334145892311961600}]
I20260928 02:36:02.001858 12118 mvcc.cc:205] Tried to move back new op lower bound from 7334145892311961600 to 7334145892153995264. Current Snapshot: MvccSnapshot[applied={T|T < 7334145892311961600}]
I20260928 02:36:02.004209 11599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:36:02.004303 11599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:36:02.005110 11599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b9d4ac93354452caa8f4a12568b589b: Sending DeleteTablet for 3 replicas of tablet d72a166640af433fb6519fa32c40f6a2
I20260928 02:36:02.005560 12018 tablet_service.cc:1514] Processing DeleteTablet for tablet d72a166640af433fb6519fa32c40f6a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:36:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49044
I20260928 02:36:02.005767 11746 tablet_service.cc:1514] Processing DeleteTablet for tablet d72a166640af433fb6519fa32c40f6a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:36:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35764
I20260928 02:36:02.006414 12124 tablet_replica.cc:330] T d72a166640af433fb6519fa32c40f6a2 P a26973bfe71a4fdca78e6a47abf8601f: stopping tablet replica
I20260928 02:36:02.006515 12124 raft_consensus.cc:2229] T d72a166640af433fb6519fa32c40f6a2 P a26973bfe71a4fdca78e6a47abf8601f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:36:02.006577 12124 raft_consensus.cc:2258] T d72a166640af433fb6519fa32c40f6a2 P a26973bfe71a4fdca78e6a47abf8601f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:36:02.006768 11883 tablet_service.cc:1514] Processing DeleteTablet for tablet d72a166640af433fb6519fa32c40f6a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:36:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38708
I20260928 02:36:02.007941 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 11681
I20260928 02:36:02.011791 12124 ts_tablet_manager.cc:1905] T d72a166640af433fb6519fa32c40f6a2 P a26973bfe71a4fdca78e6a47abf8601f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:36:02.013492 12124 ts_tablet_manager.cc:1918] T d72a166640af433fb6519fa32c40f6a2 P a26973bfe71a4fdca78e6a47abf8601f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:36:02.013634 12124 log.cc:1196] T d72a166640af433fb6519fa32c40f6a2 P a26973bfe71a4fdca78e6a47abf8601f: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-2/wal/wals/d72a166640af433fb6519fa32c40f6a2
I20260928 02:36:02.014230 12124 ts_tablet_manager.cc:1939] T d72a166640af433fb6519fa32c40f6a2 P a26973bfe71a4fdca78e6a47abf8601f: Deleting consensus metadata
I20260928 02:36:02.014693 11586 catalog_manager.cc:4841] TS a26973bfe71a4fdca78e6a47abf8601f (127.0.98.3:36399): tablet d72a166640af433fb6519fa32c40f6a2 (table test-table [id=76370d684c2c47729fee7314d831a161]) successfully deleted
I20260928 02:36:02.014926 12127 tablet_replica.cc:330] T d72a166640af433fb6519fa32c40f6a2 P 48e42310c9854ac480b1abeb10b59b77: stopping tablet replica
I20260928 02:36:02.015091 12127 raft_consensus.cc:2229] T d72a166640af433fb6519fa32c40f6a2 P 48e42310c9854ac480b1abeb10b59b77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:36:02.015245 12127 raft_consensus.cc:2258] T d72a166640af433fb6519fa32c40f6a2 P 48e42310c9854ac480b1abeb10b59b77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:36:02.015722 12127 ts_tablet_manager.cc:1905] T d72a166640af433fb6519fa32c40f6a2 P 48e42310c9854ac480b1abeb10b59b77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:36:02.017145 12127 ts_tablet_manager.cc:1918] T d72a166640af433fb6519fa32c40f6a2 P 48e42310c9854ac480b1abeb10b59b77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:36:02.017278 12127 log.cc:1196] T d72a166640af433fb6519fa32c40f6a2 P 48e42310c9854ac480b1abeb10b59b77: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790562862804661-392-0/minicluster-data/ts-1/wal/wals/d72a166640af433fb6519fa32c40f6a2
I20260928 02:36:02.017562 12127 ts_tablet_manager.cc:1939] T d72a166640af433fb6519fa32c40f6a2 P 48e42310c9854ac480b1abeb10b59b77: Deleting consensus metadata
I20260928 02:36:02.017974 11584 catalog_manager.cc:4841] TS 48e42310c9854ac480b1abeb10b59b77 (127.0.98.2:38223): tablet d72a166640af433fb6519fa32c40f6a2 (table test-table [id=76370d684c2c47729fee7314d831a161]) successfully deleted
W20260928 02:36:02.022018 11586 connection.cc:535] server connection from 127.0.98.1:34601 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 02:36:02.022225 11585 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33623 (error 108)
I20260928 02:36:02.022775 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 11818
I20260928 02:36:02.028599 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 11955
I20260928 02:36:02.029570 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:02.029448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33623 (local address 127.0.0.1:35778)
0928 02:36:02.029497 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:02.029534 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 02:36:02.029749 11585 catalog_manager.cc:4584] TS b08aacc5f2fc464d91e13586f7257802 (127.0.98.1:33623): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d72a166640af433fb6519fa32c40f6a2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33623: connect: Connection refused (error 111)
I20260928 02:36:02.034843 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 11568
I20260928 02:36:02.039341 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 02:36:02.373147 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 02:36:02.373248 392 mini_ranger.cc:66] Stopping Ranger...
I20260928 02:36:02.707288 392 mini_ranger.cc:68] Stopped Ranger
2026-09-28 02:36:02.707 UTC [11154] LOG: received smart shutdown request
2026-09-28 02:36:02.710 UTC [11154] LOG: background worker "logical replication launcher" (PID 11163) exited with exit code 1
2026-09-28 02:36:02.710 UTC [11158] LOG: shutting down
2026-09-28 02:36:02.844 UTC [11154] LOG: database system is shut down
2026-09-28T02:36:02Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (43869 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:36:02 dist-test-slave-fz47 krb5kdc[12137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:36:02 dist-test-slave-fz47 krb5kdc[12137](info): set up 2 sockets
Sep 28 02:36:02 dist-test-slave-fz47 krb5kdc[12137](info): commencing operation
krb5kdc: starting...
W20260928 02:36:03.872969 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:36:03 dist-test-slave-fz47 krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:36:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:36:03Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260928 02:36:04.019073 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskyt1071/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/postgres -l logfile start
2026-09-28 02:36:06.218 UTC [12167] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 02:36:06.218 UTC [12167] LOG: listening on IPv4 address "127.0.98.20", port 54441
2026-09-28 02:36:06.221 UTC [12167] LOG: listening on Unix socket "/tmp/.s.PGSQL.54441"
2026-09-28 02:36:06.239 UTC [12170] LOG: database system was shut down at 2026-09-28 02:36:05 UTC
2026-09-28 02:36:06.242 UTC [12167] LOG: database system is ready to accept connections
I20260928 02:36:07.100775 392 mini_postgres.cc:96] Postgres bound to 54441
2026-09-28 02:36:07.106 UTC [12178] FATAL: database "slave" does not exist
127.0.98.20:54441 - accepting connections
I20260928 02:36:07.106612 392 mini_ranger.cc:161] Starting Ranger...
I20260928 02:36:07.119803 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 02:36:07.498920 392 mini_ranger.cc:87] Created ranger Postgres database
I20260928 02:36:07.499009 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ranger-admin
2026-09-28 02:36:07,754 [I] DB FLAVOR :POSTGRES
2026-09-28 02:36:07,754 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:36:07,754 [I] Checking connection..
2026-09-28 02:36:07,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 02:36:08,003 [I] Checking connection passed.
2026-09-28 02:36:08,004 [I] --------- Verifying version history table ---------
2026-09-28 02:36:08,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:36:08,247 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:36:08,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:36:08,491 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:36:08,491 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 02:36:08,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54441/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 02:36:08.720 UTC [12242] WARNING: there is no transaction in progress
2026-09-28 02:36:08,725 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 02:36:08,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:36:08,957 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 02:36:08,958 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:36:08,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 02:36:09,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 02:36:09,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-fz47.c.gcp-upstream.internal','N') ;"
2026-09-28 02:36:09,631 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:36:09,880 [I] Table x_portal_user does not exist in database ranger
2026-09-28 02:36:09,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:36:10,123 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 02:36:10,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:36:10,378 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:36:10,604 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 02:36:10,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54441/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 02:36:11.475 UTC [12350] WARNING: there is no transaction in progress
2026-09-28 02:36:11.491 UTC [12350] WARNING: there is no transaction in progress
2026-09-28 02:36:11.506 UTC [12350] WARNING: there is no transaction in progress
2026-09-28 02:36:11.520 UTC [12350] WARNING: there is no transaction in progress
2026-09-28 02:36:11.534 UTC [12350] WARNING: there is no transaction in progress
2026-09-28 02:36:11.550 UTC [12350] WARNING: there is no transaction in progress
2026-09-28 02:36:11.600 UTC [12350] WARNING: there is no transaction in progress
2026-09-28 02:36:11.605 UTC [12350] WARNING: there is no transaction in progress
2026-09-28 02:36:11.618 UTC [12350] WARNING: there is no transaction in progress
2026-09-28 02:36:11.626 UTC [12350] WARNING: there is no transaction in progress
2026-09-28 02:36:11.635 UTC [12350] WARNING: there is no transaction in progress
2026-09-28 02:36:11.648 UTC [12350] WARNING: there is no transaction in progress
2026-09-28 02:36:11.656 UTC [12350] WARNING: there is no transaction in progress
2026-09-28 02:36:11.662 UTC [12350] WARNING: there is no transaction in progress
2026-09-28 02:36:11.667 UTC [12350] WARNING: there is no transaction in progress
2026-09-28 02:36:11.673 UTC [12350] WARNING: there is no transaction in progress
2026-09-28 02:36:12,211 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 02:36:12,211 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 02:36:12,452 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 02:36:12,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:36:12,674 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 02:36:12,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:36:12,910 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 02:36:12,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:36:13,162 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:36:13,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-fz47.c.gcp-upstream.internal';"
2026-09-28 02:36:13,643 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 02:36:13,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54441/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 02:36:13,887 [I] DB_PATCHES have already been applied
I20260928 02:36:13.892691 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskyt1071/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 02:36:13.892748 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260928 02:36:13.895378 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:36625
Sep 28, 2026 2:36:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:36:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:36:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:36:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:36:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:36:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:36:14 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 02:36:14.888026 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.781s user 0.000s sys 0.007s
Sep 28, 2026 2:36:34 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:36:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 2:36:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:36:34 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:36:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:36:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:36:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:36:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:36:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:36:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:36:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:36:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:36:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:36:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:36:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:36:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:36:34 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:36:35 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 02:36:35.707480 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 02:36:35 dist-test-slave-fz47 krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562995, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 02:36:35.780740 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 02:36:35.789067 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 02:36:36.501585 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 02:36:36.501664 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ranger-kms
I20260928 02:36:36.501678 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:54441
2026-09-28 02:36:36,644 [I] DB FLAVOR :POSTGRES
2026-09-28 02:36:36,644 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:36:36,644 [I] Checking connection
2026-09-28 02:36:36,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54441/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 02:36:36,871 [I] connection success
2026-09-28 02:36:36,872 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 02:36:36,872 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 02:36:36,872 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54441/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 02:36:37,098 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 02:36:37,098 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:36:37,099 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 02:36:37,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54441/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 02:36:37,344 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 02:36:37.799717 392 mini_ranger_kms.cc:310] Created kms service
I20260928 02:36:37.989655 392 mini_ranger_kms.cc:326] Created kudu user
I20260928 02:36:38.029407 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 02:36:38 dist-test-slave-fz47 krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790562995, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 02:36:38.386263 392 mini_ranger_kms.cc:384] Added ranger policy
I20260928 02:36:38.386380 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskyt1071/build/release/bin/postgresql.jar:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskyt1071/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 02:36:38.386400 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260928 02:36:38.389760 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12530
I20260928 02:36:38.389822 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:46731
Sep 28, 2026 2:36:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:36:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:36:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:36:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:36:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:36:39 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 02:36:40.415400 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.635s user 0.001s sys 0.125s
I20260928 02:36:40.415480 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 02:36:40.415526 392 mini_ranger_kms.cc:401] 127.0.98.20:46731/kms/v1/keys
Sep 28 02:36:40 dist-test-slave-fz47 krb5kdc[12137](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790563000, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 02:36:40 dist-test-slave-fz47 krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563000, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 02:36:41 dist-test-slave-fz47 krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563000, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 02:36:43 dist-test-slave-fz47 krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 02:36:43.490012 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38263
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38263
--ranger_config_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46731/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:36:43.567698 12581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:36:43.567901 12581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:36:43.567921 12581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:36:43.569021 12581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:36:43.569061 12581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:36:43.569079 12581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:36:43.569095 12581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:36:43.569151 12581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:36:43.570784 12581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:46731/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38263
--ranger_config_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:36:43.570997 12581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:36:43.571197 12581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:36:43.575793 12589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:43.575891 12581 server_base.cc:1034] running on GCE node
W20260928 02:36:43.575824 12587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:43.576040 12586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:43.576305 12581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:36:43.576565 12581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:36:43.577697 12581 hybrid_clock.cc:648] HybridClock initialized: now 1790563003577691 us; error 36 us; skew 500 ppm
Sep 28 02:36:43 dist-test-slave-fz47 krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563003, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:36:43.580147 12581 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:36:43.580569 12581 webserver.cc:460] Webserver started at http://127.0.98.62:37803/ using document root <none> and password file <none>
I20260928 02:36:43.580761 12581 fs_manager.cc:359] Metadata directory not provided
I20260928 02:36:43.580801 12581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:36:43.580910 12581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:36:43 dist-test-slave-fz47 krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563003, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 02:36:43.625942 12581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "b7c2f9d26ec541b4841a29fbe01d8841"
format_stamp: "Formatted at 2026-09-28 02:36:43 on dist-test-slave-fz47"
server_key: "3dc9d844cdbba074bf7026d4b972f944"
server_key_iv: "e784759128590c21f58f430af1e4e508"
server_key_version: "kuduclusterkey@0"
I20260928 02:36:43.626328 12581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "b7c2f9d26ec541b4841a29fbe01d8841"
format_stamp: "Formatted at 2026-09-28 02:36:43 on dist-test-slave-fz47"
server_key: "3dc9d844cdbba074bf7026d4b972f944"
server_key_iv: "e784759128590c21f58f430af1e4e508"
server_key_version: "kuduclusterkey@0"
I20260928 02:36:43.627627 12581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 02:36:43.638968 12596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:43.639191 12581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:36:43.639304 12581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "b7c2f9d26ec541b4841a29fbe01d8841"
format_stamp: "Formatted at 2026-09-28 02:36:43 on dist-test-slave-fz47"
server_key: "3dc9d844cdbba074bf7026d4b972f944"
server_key_iv: "e784759128590c21f58f430af1e4e508"
server_key_version: "kuduclusterkey@0"
I20260928 02:36:43.639396 12581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:36:43.667790 12581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:36:43.669808 12581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:36:43.669951 12581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:36:43.674520 12581 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38263
I20260928 02:36:43.674552 12648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38263 every 8 connection(s)
I20260928 02:36:43.674937 12581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:36:43.675632 12649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:36:43.677904 12649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841: Bootstrap starting.
I20260928 02:36:43.678460 12649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841: Neither blocks nor log segments found. Creating new log.
I20260928 02:36:43.678725 12649 log.cc:824] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841: Log is configured to *not* fsync() on all Append() calls
I20260928 02:36:43.679314 12649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841: No bootstrap required, opened a new log
I20260928 02:36:43.681324 12649 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7c2f9d26ec541b4841a29fbe01d8841" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38263 } }
I20260928 02:36:43.681470 12649 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:36:43.681502 12649 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7c2f9d26ec541b4841a29fbe01d8841, State: Initialized, Role: FOLLOWER
I20260928 02:36:43.681648 12649 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7c2f9d26ec541b4841a29fbe01d8841" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38263 } }
I20260928 02:36:43.681721 12649 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:36:43.681749 12649 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:36:43.681794 12649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:36:43.682296 12649 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7c2f9d26ec541b4841a29fbe01d8841" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38263 } }
I20260928 02:36:43.682384 12649 leader_election.cc:302] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7c2f9d26ec541b4841a29fbe01d8841; no voters:
I20260928 02:36:43.682528 12649 leader_election.cc:288] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:36:43.682564 12652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:36:43.682678 12652 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [term 1 LEADER]: Becoming Leader. State: Replica: b7c2f9d26ec541b4841a29fbe01d8841, State: Running, Role: LEADER
I20260928 02:36:43.682794 12649 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:36:43.682812 12652 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7c2f9d26ec541b4841a29fbe01d8841" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38263 } }
I20260928 02:36:43.683146 12652 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7c2f9d26ec541b4841a29fbe01d8841. Latest consensus state: current_term: 1 leader_uuid: "b7c2f9d26ec541b4841a29fbe01d8841" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7c2f9d26ec541b4841a29fbe01d8841" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38263 } } }
I20260928 02:36:43.683226 12652 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [sys.catalog]: This master's current role is: LEADER
I20260928 02:36:43.683154 12653 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7c2f9d26ec541b4841a29fbe01d8841" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7c2f9d26ec541b4841a29fbe01d8841" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38263 } } }
I20260928 02:36:43.683336 12653 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [sys.catalog]: This master's current role is: LEADER
I20260928 02:36:43.683600 12658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:36:43.684068 12658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:36:43.684396 12649 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 02:36:43.685111 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 12581
I20260928 02:36:43.685199 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:36:43.685387 392 external_mini_cluster.cc:1428] Setting key 17e3f26ee7918a5e955a0cfe9358d36e
I20260928 02:36:43.690001 12658 catalog_manager.cc:1347] Generated new cluster ID: d6764fd07a1744c785e40934944d3290
I20260928 02:36:43.690109 12658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:36:43 dist-test-slave-fz47 krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:36:43.704790 12672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:43.687910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51404 (local address 127.0.98.62:38263)
0928 02:36:43.693421 (+ 5511us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:43.693433 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:43.693477 (+ 44us) server_negotiation.cc:407] Connection header received
0928 02:36:43.693717 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:43.693725 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:43.693789 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:43.694711 (+ 922us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:43.694893 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:43.696396 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:43.697523 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:43.698218 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:43.698636 (+ 418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:43.700998 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:43.701002 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:43.701002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:43.701014 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:43.702907 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:43.703278 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:43.703281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:43.703281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:43.703372 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:43.703566 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:43.703567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:43.703567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:43.703863 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:43.704380 (+ 517us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:43.704447 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 02:36:43.704561 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5437,"thread_start_us":227,"threads_started":1}
I20260928 02:36:43.713325 12658 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:36:43.714278 12658 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:36:43.721346 12658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841: Generated new TSK 0
I20260928 02:36:43.721627 12658 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 02:36:44.466173 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46731/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38263
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:36:44.574955 12694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:36:44.575145 12694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:36:44.575174 12694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:36:44.576295 12694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:36:44.576337 12694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:36:44.576372 12694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:36:44.577814 12694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:46731/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:36:44.578024 12694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:36:44.578224 12694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:36:44.582826 12700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:44.582829 12699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:44.583081 12702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:44.583595 12694 server_base.cc:1034] running on GCE node
I20260928 02:36:44.583794 12694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:36:44.584084 12694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:36:44.585239 12694 hybrid_clock.cc:648] HybridClock initialized: now 1790563004585221 us; error 30 us; skew 500 ppm
Sep 28 02:36:44 dist-test-slave-fz47 krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:36:44.588299 12694 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 02:36:44.588920 12694 webserver.cc:460] Webserver started at http://127.0.98.1:42311/ using document root <none> and password file <none>
I20260928 02:36:44.589233 12694 fs_manager.cc:359] Metadata directory not provided
I20260928 02:36:44.589326 12694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:36:44.589465 12694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:36:44 dist-test-slave-fz47 krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 02:36:44.604077 12694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-0/data/instance:
uuid: "0676e01e15d245c6878a6d3d093d43da"
format_stamp: "Formatted at 2026-09-28 02:36:44 on dist-test-slave-fz47"
server_key: "2d126be4a4831f9e3d137d39f8468937"
server_key_iv: "1a2893c3726b6e6f19018dc2e273370f"
server_key_version: "kuduclusterkey@0"
I20260928 02:36:44.604516 12694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0676e01e15d245c6878a6d3d093d43da"
format_stamp: "Formatted at 2026-09-28 02:36:44 on dist-test-slave-fz47"
server_key: "2d126be4a4831f9e3d137d39f8468937"
server_key_iv: "1a2893c3726b6e6f19018dc2e273370f"
server_key_version: "kuduclusterkey@0"
I20260928 02:36:44.609161 12694 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 02:36:44.622460 12711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:44.622637 12694 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 02:36:44.622721 12694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "0676e01e15d245c6878a6d3d093d43da"
format_stamp: "Formatted at 2026-09-28 02:36:44 on dist-test-slave-fz47"
server_key: "2d126be4a4831f9e3d137d39f8468937"
server_key_iv: "1a2893c3726b6e6f19018dc2e273370f"
server_key_version: "kuduclusterkey@0"
I20260928 02:36:44.622819 12694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:36:44.634586 12694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:36:44.636765 12694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:36:44.636951 12694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:36:44.637507 12694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:36:44.637897 12694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:36:44.637980 12694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:44.638072 12694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:36:44.638140 12694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:44.644197 12694 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39979
I20260928 02:36:44.644740 12694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
I20260928 02:36:44.646864 12824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39979 every 8 connection(s)
I20260928 02:36:44.651679 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 12694
I20260928 02:36:44.651777 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-0/wal/instance
I20260928 02:36:44.651944 392 external_mini_cluster.cc:1428] Setting key 073841ce8ea935b417395713d26ca31d
Sep 28 02:36:44 dist-test-slave-fz47 krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:36:44.657089 12828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:44.646532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57921 (local address 127.0.98.62:38263)
0928 02:36:44.646705 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:44.646707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:44.646890 (+ 183us) server_negotiation.cc:407] Connection header received
0928 02:36:44.647566 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:44.647567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:44.647582 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:44.647671 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:44.648448 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:44.649287 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:44.650757 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:44.651061 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:44.651130 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:44.653952 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:44.653953 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:44.653953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:44.653961 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:44.655595 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:44.656073 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:44.656075 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:44.656075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:44.656147 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:44.656400 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:44.656401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:44.656401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:44.656533 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:44.656599 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:44.656911 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 02:36:44.656991 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":77,"threads_started":1}
I20260928 02:36:44.657238 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:44.646258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38263 (local address 127.0.98.1:57921)
0928 02:36:44.646769 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:44.646813 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:44.647440 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:44.647812 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:44.647816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:44.648052 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:36:44.648331 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:44.648338 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:44.649386 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:44.649387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:44.650662 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:44.650665 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:44.651171 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:44.651172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:44.651310 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:44.652322 (+ 1012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:36:44.652331 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:36:44.653859 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:36:44.655685 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:44.655691 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:44.655693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:44.655984 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:44.656207 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:44.656208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:44.656208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:44.656339 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:44.656600 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:36:44.656605 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:36:44.656771 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 02:36:44.656882 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 02:36:44.657021 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":84,"threads_started":1}
I20260928 02:36:44.658105 12825 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38263
I20260928 02:36:44.658192 12825 heartbeater.cc:464] Registering TS with master...
I20260928 02:36:44.658366 12825 heartbeater.cc:511] Master 127.0.98.62:38263 requested a full tablet report, sending...
I20260928 02:36:44.658895 12613 ts_manager.cc:194] Registered new tserver with Master: 0676e01e15d245c6878a6d3d093d43da (127.0.98.1:39979)
I20260928 02:36:44.659756 12613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57921
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 02:36:44.675581 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46731/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38263
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:36:44 dist-test-slave-fz47 krb5kdc[12137](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790563004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 02:36:44.788359 12831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:36:44.788650 12831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:36:44.788689 12831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:36:44.790460 12831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:36:44.790524 12831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:36:44.790563 12831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:36:44.792695 12831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:46731/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:36:44.792953 12831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:36:44.793207 12831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:36:44.799367 12841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:44.799618 12839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:44.799787 12838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:44.800951 12831 server_base.cc:1034] running on GCE node
I20260928 02:36:44.801175 12831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:36:44.801469 12831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:36:44.802650 12831 hybrid_clock.cc:648] HybridClock initialized: now 1790563004802622 us; error 39 us; skew 500 ppm
Sep 28 02:36:44 dist-test-slave-fz47 krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:36:44.805732 12831 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 02:36:44.806279 12831 webserver.cc:460] Webserver started at http://127.0.98.2:36531/ using document root <none> and password file <none>
I20260928 02:36:44.806523 12831 fs_manager.cc:359] Metadata directory not provided
I20260928 02:36:44.806591 12831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:36:44.806730 12831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:36:44 dist-test-slave-fz47 krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 02:36:44.821467 12831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-1/data/instance:
uuid: "c12d1288231445a8844827cd2e10eca5"
format_stamp: "Formatted at 2026-09-28 02:36:44 on dist-test-slave-fz47"
server_key: "8baa76b34d02314f1d5795a4ffb82ba1"
server_key_iv: "26285592103eefd3f8e0d1b073fc3c70"
server_key_version: "kuduclusterkey@0"
I20260928 02:36:44.821993 12831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c12d1288231445a8844827cd2e10eca5"
format_stamp: "Formatted at 2026-09-28 02:36:44 on dist-test-slave-fz47"
server_key: "8baa76b34d02314f1d5795a4ffb82ba1"
server_key_iv: "26285592103eefd3f8e0d1b073fc3c70"
server_key_version: "kuduclusterkey@0"
I20260928 02:36:44.825767 12831 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:36:44.849334 12848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:44.849581 12831 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.001s sys 0.000s
I20260928 02:36:44.849689 12831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "c12d1288231445a8844827cd2e10eca5"
format_stamp: "Formatted at 2026-09-28 02:36:44 on dist-test-slave-fz47"
server_key: "8baa76b34d02314f1d5795a4ffb82ba1"
server_key_iv: "26285592103eefd3f8e0d1b073fc3c70"
server_key_version: "kuduclusterkey@0"
I20260928 02:36:44.849807 12831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:36:44.868839 12831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:36:44.871145 12831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:36:44.871284 12831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:36:44.871512 12831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:36:44.871872 12831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:36:44.871932 12831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:44.871961 12831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:36:44.871977 12831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:44.878562 12831 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41537
I20260928 02:36:44.878983 12831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
I20260928 02:36:44.881739 12962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41537 every 8 connection(s)
I20260928 02:36:44.881762 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 12831
I20260928 02:36:44.881855 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-1/wal/instance
I20260928 02:36:44.882045 392 external_mini_cluster.cc:1428] Setting key a1805c9967281b65377dbf8ed592018b
Sep 28 02:36:44 dist-test-slave-fz47 krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:36:44.892856 12828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:44.882723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38393 (local address 127.0.98.62:38263)
0928 02:36:44.882763 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:44.882765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:44.882892 (+ 127us) server_negotiation.cc:407] Connection header received
0928 02:36:44.883642 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:44.883643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:44.883656 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:44.883685 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:44.884491 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:44.885338 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:44.886811 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:44.887116 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:44.887207 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:44.889638 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:44.889640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:44.889640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:44.889648 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:44.891230 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:44.891761 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:44.891762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:44.891762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:44.891826 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:44.892113 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:44.892114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:44.892114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:44.892250 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:44.892315 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:44.892665 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 02:36:44.892756 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 02:36:44.893047 12965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:44.882308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38263 (local address 127.0.98.2:38393)
0928 02:36:44.882724 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:44.882811 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:44.883489 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:44.883871 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:44.883875 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:44.884115 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:36:44.884384 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:44.884392 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:44.885444 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:44.885445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:44.886718 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:44.886721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:44.887226 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:44.887227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:44.887363 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:44.887951 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:36:44.887963 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:36:44.889535 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:36:44.891321 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:44.891329 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:44.891331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:44.891667 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:44.891891 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:44.891892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:44.891892 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:44.892042 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:44.892321 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:36:44.892327 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:36:44.892506 (+ 179us) client_negotiation.cc:769] Sending connection context
0928 02:36:44.892636 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 02:36:44.892800 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":84,"threads_started":1}
I20260928 02:36:44.893822 12963 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38263
I20260928 02:36:44.893926 12963 heartbeater.cc:464] Registering TS with master...
I20260928 02:36:44.894114 12963 heartbeater.cc:511] Master 127.0.98.62:38263 requested a full tablet report, sending...
I20260928 02:36:44.894461 12613 ts_manager.cc:194] Registered new tserver with Master: c12d1288231445a8844827cd2e10eca5 (127.0.98.2:41537)
I20260928 02:36:44.895066 12613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38393
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 02:36:44.920346 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46731/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38263
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:36:45.030395 12968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:36:45.030603 12968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:36:45.030623 12968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:36:45.031785 12968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:36:45.031827 12968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:36:45.031852 12968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:36:45.033378 12968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:46731/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:36:45.033592 12968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:36:45.033797 12968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:36:45.038509 12976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:45.038522 12974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:45.038509 12973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:45.038664 12968 server_base.cc:1034] running on GCE node
I20260928 02:36:45.038968 12968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:36:45.039191 12968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:36:45.040333 12968 hybrid_clock.cc:648] HybridClock initialized: now 1790563005040317 us; error 29 us; skew 500 ppm
Sep 28 02:36:45 dist-test-slave-fz47 krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:36:45.042989 12968 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 02:36:45.043424 12968 webserver.cc:460] Webserver started at http://127.0.98.3:40997/ using document root <none> and password file <none>
I20260928 02:36:45.043654 12968 fs_manager.cc:359] Metadata directory not provided
I20260928 02:36:45.043697 12968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:36:45.043814 12968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:36:45 dist-test-slave-fz47 krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 02:36:45.055580 12968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-2/data/instance:
uuid: "9f063b3987a34ea499de06b3ef2e0fca"
format_stamp: "Formatted at 2026-09-28 02:36:45 on dist-test-slave-fz47"
server_key: "f1d527a5c1e5cfb7186b5035e70539fd"
server_key_iv: "825d2c4371cf31727a414e4f68963cc0"
server_key_version: "kuduclusterkey@0"
I20260928 02:36:45.056561 12968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9f063b3987a34ea499de06b3ef2e0fca"
format_stamp: "Formatted at 2026-09-28 02:36:45 on dist-test-slave-fz47"
server_key: "f1d527a5c1e5cfb7186b5035e70539fd"
server_key_iv: "825d2c4371cf31727a414e4f68963cc0"
server_key_version: "kuduclusterkey@0"
I20260928 02:36:45.063269 12968 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.001s
I20260928 02:36:45.089121 12983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:45.089322 12968 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260928 02:36:45.089424 12968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "9f063b3987a34ea499de06b3ef2e0fca"
format_stamp: "Formatted at 2026-09-28 02:36:45 on dist-test-slave-fz47"
server_key: "f1d527a5c1e5cfb7186b5035e70539fd"
server_key_iv: "825d2c4371cf31727a414e4f68963cc0"
server_key_version: "kuduclusterkey@0"
I20260928 02:36:45.089516 12968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:36:45.104981 12968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:36:45.107318 12968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:36:45.107477 12968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:36:45.107700 12968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:36:45.108079 12968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:36:45.108139 12968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:45.108174 12968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:36:45.108196 12968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:45.114749 12968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44937
I20260928 02:36:45.114830 13096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44937 every 8 connection(s)
I20260928 02:36:45.115185 12968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
I20260928 02:36:45.116709 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 12968
I20260928 02:36:45.116796 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-2/wal/instance
I20260928 02:36:45.116981 392 external_mini_cluster.cc:1428] Setting key dbff0d8febcfe59d32417a1fcd2f13d7
Sep 28 02:36:45 dist-test-slave-fz47 krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:36:45.127269 12828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:45.116605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48883 (local address 127.0.98.62:38263)
0928 02:36:45.116661 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:45.116664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:45.117046 (+ 382us) server_negotiation.cc:407] Connection header received
0928 02:36:45.117924 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:45.117924 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:45.117935 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:45.117974 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:45.118699 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:45.119483 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:45.120906 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:45.121238 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:45.121331 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:45.123824 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:45.123826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:45.123826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:45.123833 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:45.125635 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:45.126108 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:45.126109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:45.126109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:45.126172 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:45.126475 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:45.126476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:45.126476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:45.126649 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:45.126712 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:45.127080 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 02:36:45.127169 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 02:36:45.127455 13099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:45.116590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38263 (local address 127.0.98.3:48883)
0928 02:36:45.116908 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:45.116956 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:45.117669 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:45.118064 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:45.118068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:45.118309 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:36:45.118598 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:45.118605 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:45.119590 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:45.119591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:45.120811 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:45.120813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:45.121348 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:45.121349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:45.121516 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:45.122078 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:36:45.122088 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:36:45.123720 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:36:45.125727 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:45.125734 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:45.125737 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:45.126031 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:45.126243 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:45.126244 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:45.126244 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:45.126402 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:45.126704 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:36:45.126710 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:36:45.126896 (+ 186us) client_negotiation.cc:769] Sending connection context
0928 02:36:45.127048 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 02:36:45.127205 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":148,"threads_started":1}
I20260928 02:36:45.128029 13097 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38263
I20260928 02:36:45.128127 13097 heartbeater.cc:464] Registering TS with master...
I20260928 02:36:45.128304 13097 heartbeater.cc:511] Master 127.0.98.62:38263 requested a full tablet report, sending...
I20260928 02:36:45.128691 12613 ts_manager.cc:194] Registered new tserver with Master: 9f063b3987a34ea499de06b3ef2e0fca (127.0.98.3:44937)
I20260928 02:36:45.129177 12613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48883
I20260928 02:36:45.129501 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:36:45.130167 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 12694
I20260928 02:36:45.134965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 12831
I20260928 02:36:45.139825 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 12968
I20260928 02:36:45.144695 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 12581
I20260928 02:36:45.149199 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38263
--webserver_interface=127.0.98.62
--webserver_port=37803
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38263
--ranger_config_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46731/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46731/kms with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:36:45.226715 13100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:36:45.226922 13100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:36:45.226943 13100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:36:45.228087 13100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:36:45.228132 13100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:36:45.228145 13100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:36:45.228155 13100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:36:45.228166 13100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:36:45.229780 13100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46731/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38263
--ranger_config_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:36:45.230001 13100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:36:45.230219 13100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:36:45.235127 13105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:45.235108 13106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:45.235107 13108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:45.235137 13100 server_base.cc:1034] running on GCE node
I20260928 02:36:45.235687 13100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:36:45.235986 13100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:36:45.237144 13100 hybrid_clock.cc:648] HybridClock initialized: now 1790563005237112 us; error 59 us; skew 500 ppm
Sep 28 02:36:45 dist-test-slave-fz47 krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:36:45.239707 13100 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:36:45.240134 13100 webserver.cc:460] Webserver started at http://127.0.98.62:37803/ using document root <none> and password file <none>
I20260928 02:36:45.240350 13100 fs_manager.cc:359] Metadata directory not provided
I20260928 02:36:45.240415 13100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:36:45 dist-test-slave-fz47 krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 02:36:45.296818 13100 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:36:45.297693 13116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:45.297887 13100 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 02:36:45.297982 13100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "b7c2f9d26ec541b4841a29fbe01d8841"
format_stamp: "Formatted at 2026-09-28 02:36:43 on dist-test-slave-fz47"
server_key: "3dc9d844cdbba074bf7026d4b972f944"
server_key_iv: "e784759128590c21f58f430af1e4e508"
server_key_version: "kuduclusterkey@0"
I20260928 02:36:45.298324 13100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:36:45.316624 13100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:36:45.318190 13100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:36:45.318305 13100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:36:45.322207 13100 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38263
I20260928 02:36:45.322256 13168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38263 every 8 connection(s)
I20260928 02:36:45.322616 13100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:36:45.323788 13169 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:36:45.324146 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 13100
I20260928 02:36:45.324410 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46731/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39979
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38263
--webserver_port=42311
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46731/kms with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:36:45.324602 13169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841: Bootstrap starting.
I20260928 02:36:45.332614 13169 log.cc:824] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841: Log is configured to *not* fsync() on all Append() calls
I20260928 02:36:45.334360 13169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:36:45.334612 13169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841: Bootstrap complete.
I20260928 02:36:45.336752 13169 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7c2f9d26ec541b4841a29fbe01d8841" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38263 } }
I20260928 02:36:45.336985 13169 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7c2f9d26ec541b4841a29fbe01d8841, State: Initialized, Role: FOLLOWER
I20260928 02:36:45.337145 13169 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7c2f9d26ec541b4841a29fbe01d8841" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38263 } }
I20260928 02:36:45.337212 13169 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:36:45.337242 13169 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:36:45.337308 13169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:36:45.338346 13169 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7c2f9d26ec541b4841a29fbe01d8841" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38263 } }
I20260928 02:36:45.338442 13169 leader_election.cc:302] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7c2f9d26ec541b4841a29fbe01d8841; no voters:
I20260928 02:36:45.338593 13169 leader_election.cc:288] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:36:45.338647 13173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:36:45.338763 13173 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [term 2 LEADER]: Becoming Leader. State: Replica: b7c2f9d26ec541b4841a29fbe01d8841, State: Running, Role: LEADER
I20260928 02:36:45.338857 13169 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:36:45.338846 13173 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7c2f9d26ec541b4841a29fbe01d8841" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38263 } }
I20260928 02:36:45.339205 13174 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b7c2f9d26ec541b4841a29fbe01d8841" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7c2f9d26ec541b4841a29fbe01d8841" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38263 } } }
I20260928 02:36:45.339217 13175 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7c2f9d26ec541b4841a29fbe01d8841. Latest consensus state: current_term: 2 leader_uuid: "b7c2f9d26ec541b4841a29fbe01d8841" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7c2f9d26ec541b4841a29fbe01d8841" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38263 } } }
I20260928 02:36:45.339376 13175 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [sys.catalog]: This master's current role is: LEADER
I20260928 02:36:45.339357 13174 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841 [sys.catalog]: This master's current role is: LEADER
I20260928 02:36:45.339797 13183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:36:45.340289 13183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:36:45.340449 13183 catalog_manager.cc:1259] Loaded cluster ID: d6764fd07a1744c785e40934944d3290
I20260928 02:36:45.340515 13183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:36:45.340607 13169 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 02:36:45.343954 13183 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:36:45.344530 13183 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841: Loaded TSK: 0
I20260928 02:36:45.344846 13183 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:36:45.432510 13171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:36:45.432698 13171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:36:45.432726 13171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:36:45.434051 13171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:36:45.434095 13171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:36:45.434129 13171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:36:45.435525 13171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46731/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42311
--tserver_master_addrs=127.0.98.62:38263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:36:45.435750 13171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:36:45.435977 13171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:36:45.440932 13209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:45.440970 13206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:45.441160 13207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:45.441432 13171 server_base.cc:1034] running on GCE node
I20260928 02:36:45.441620 13171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:36:45.441839 13171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:36:45.442988 13171 hybrid_clock.cc:648] HybridClock initialized: now 1790563005442953 us; error 48 us; skew 500 ppm
Sep 28 02:36:45 dist-test-slave-fz47 krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:36:45.445364 13171 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 02:36:45.445839 13171 webserver.cc:460] Webserver started at http://127.0.98.1:42311/ using document root <none> and password file <none>
I20260928 02:36:45.446048 13171 fs_manager.cc:359] Metadata directory not provided
I20260928 02:36:45.446105 13171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:36:45 dist-test-slave-fz47 krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 02:36:45.464911 13171 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260928 02:36:45.465605 13217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:45.465785 13171 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:45.465871 13171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "0676e01e15d245c6878a6d3d093d43da"
format_stamp: "Formatted at 2026-09-28 02:36:44 on dist-test-slave-fz47"
server_key: "2d126be4a4831f9e3d137d39f8468937"
server_key_iv: "1a2893c3726b6e6f19018dc2e273370f"
server_key_version: "kuduclusterkey@0"
I20260928 02:36:45.466251 13171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:36:45.487968 13171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:36:45.490227 13171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:36:45.490378 13171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:36:45.490629 13171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:36:45.490998 13171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:36:45.491050 13171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:45.491089 13171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:36:45.491112 13171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:45.497623 13171 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39979
I20260928 02:36:45.498035 13171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
I20260928 02:36:45.499262 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 13171
I20260928 02:36:45.499541 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46731/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41537
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38263
--webserver_port=36531
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46731/kms with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:36:45.505578 13330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39979 every 8 connection(s)
Sep 28 02:36:45 dist-test-slave-fz47 krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:36:45.521790 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:45.506660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42017 (local address 127.0.98.62:38263)
0928 02:36:45.510051 (+ 3391us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:45.510060 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:45.510101 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:36:45.510221 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:45.510226 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:45.510266 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:45.510376 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:45.510958 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:45.512063 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:45.513024 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:45.513532 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:45.513629 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:45.518607 (+ 4978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:45.518611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:45.518612 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:45.518621 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:45.520093 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:45.520484 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:45.520487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:45.520487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:45.520594 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:45.520811 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:45.520813 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:45.520813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:45.521094 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:45.521172 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:45.521485 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 02:36:45.521584 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3341,"thread_start_us":74,"threads_started":1}
I20260928 02:36:45.521832 13334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:45.506264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38263 (local address 127.0.98.1:42017)
0928 02:36:45.509160 (+ 2896us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:45.509210 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:45.509854 (+ 644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:45.510438 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:45.510442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:45.510633 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:36:45.510818 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:45.510823 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:45.512161 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:45.512162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:45.512945 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:45.512947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:45.513729 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:45.513730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:45.513855 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:45.517311 (+ 3456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:36:45.517320 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:36:45.518505 (+ 1185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:36:45.520170 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:45.520176 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:45.520178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:45.520398 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:45.520658 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:45.520659 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:45.520659 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:45.520754 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:45.521161 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:36:45.521168 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:36:45.521328 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 02:36:45.521425 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 02:36:45.521578 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2842,"thread_start_us":99,"threads_started":1}
I20260928 02:36:45.522796 13331 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38263
I20260928 02:36:45.522888 13331 heartbeater.cc:464] Registering TS with master...
I20260928 02:36:45.523058 13331 heartbeater.cc:511] Master 127.0.98.62:38263 requested a full tablet report, sending...
W20260928 02:36:45.523799 13331 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38263 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:36:45.523857 13331 heartbeater.cc:464] Registering TS with master...
I20260928 02:36:45.523917 13331 heartbeater.cc:511] Master 127.0.98.62:38263 requested a full tablet report, sending...
I20260928 02:36:45.524219 13331 heartbeater.cc:464] Registering TS with master...
I20260928 02:36:45.524287 13331 heartbeater.cc:511] Master 127.0.98.62:38263 requested a full tablet report, sending...
W20260928 02:36:45.524533 13331 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:36:45.616683 13333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:36:45.616976 13333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:36:45.617018 13333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:36:45.618700 13333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:36:45.618757 13333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:36:45.618794 13333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:36:45.620963 13333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46731/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41537
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36531
--tserver_master_addrs=127.0.98.62:38263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:36:45.621212 13333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:36:45.621503 13333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:36:45.627749 13343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:45.627900 13340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:45.627959 13341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:45.628116 13333 server_base.cc:1034] running on GCE node
I20260928 02:36:45.628304 13333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:36:45.628559 13333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:36:45.629703 13333 hybrid_clock.cc:648] HybridClock initialized: now 1790563005629686 us; error 32 us; skew 500 ppm
Sep 28 02:36:45 dist-test-slave-fz47 krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:36:45.632028 13333 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 02:36:45.632505 13333 webserver.cc:460] Webserver started at http://127.0.98.2:36531/ using document root <none> and password file <none>
I20260928 02:36:45.632738 13333 fs_manager.cc:359] Metadata directory not provided
I20260928 02:36:45.632802 13333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:36:45 dist-test-slave-fz47 krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 02:36:45.649685 13333 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:36:45.650346 13351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:45.650507 13333 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:45.650583 13333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "c12d1288231445a8844827cd2e10eca5"
format_stamp: "Formatted at 2026-09-28 02:36:44 on dist-test-slave-fz47"
server_key: "8baa76b34d02314f1d5795a4ffb82ba1"
server_key_iv: "26285592103eefd3f8e0d1b073fc3c70"
server_key_version: "kuduclusterkey@0"
I20260928 02:36:45.650923 13333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:36:45.662192 13333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:36:45.664031 13333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:36:45.664157 13333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:36:45.664345 13333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:36:45.664670 13333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:36:45.664705 13333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:45.664732 13333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:36:45.664753 13333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:45.670948 13333 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41537
I20260928 02:36:45.671391 13333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
I20260928 02:36:45.671324 13464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41537 every 8 connection(s)
Sep 28 02:36:45 dist-test-slave-fz47 krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:36:45.680045 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 13333
I20260928 02:36:45.680336 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46731/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44937
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38263
--webserver_port=40997
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46731/kms with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:36:45.681250 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:45.672164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39507 (local address 127.0.98.62:38263)
0928 02:36:45.672212 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:45.672214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:45.672439 (+ 225us) server_negotiation.cc:407] Connection header received
0928 02:36:45.673071 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:45.673072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:45.673081 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:45.673122 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:45.673827 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:45.674626 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:45.676074 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:45.676266 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:45.676321 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:45.678424 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:45.678425 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:45.678425 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:45.678432 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:45.679641 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:45.680140 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:45.680142 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:45.680142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:45.680211 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:45.680508 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:45.680509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:45.680509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:45.680641 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:45.680712 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:45.681077 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 02:36:45.681160 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 02:36:45.681473 13467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:45.672135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38263 (local address 127.0.98.2:39507)
0928 02:36:45.672299 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:45.672346 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:45.672973 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:45.673204 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:45.673209 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:45.673456 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:36:45.673724 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:45.673731 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:45.674729 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:45.674730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:45.675986 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:45.675989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:45.676364 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:45.676365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:45.676501 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:45.676948 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:36:45.676958 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:36:45.678337 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:36:45.679724 (+ 1387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:45.679730 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:45.679732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:45.680041 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:45.680290 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:45.680290 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:45.680291 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:45.680431 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:45.680713 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:36:45.680719 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:36:45.680901 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 02:36:45.681053 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 02:36:45.681217 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260928 02:36:45.681947 13465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38263
I20260928 02:36:45.682044 13465 heartbeater.cc:464] Registering TS with master...
I20260928 02:36:45.682219 13465 heartbeater.cc:511] Master 127.0.98.62:38263 requested a full tablet report, sending...
W20260928 02:36:45.682855 13465 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38263 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:36:45.682925 13465 heartbeater.cc:464] Registering TS with master...
I20260928 02:36:45.682998 13465 heartbeater.cc:511] Master 127.0.98.62:38263 requested a full tablet report, sending...
I20260928 02:36:45.683352 13465 heartbeater.cc:464] Registering TS with master...
I20260928 02:36:45.683423 13465 heartbeater.cc:511] Master 127.0.98.62:38263 requested a full tablet report, sending...
W20260928 02:36:45.683735 13465 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:36:45.789942 13468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:36:45.790227 13468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:36:45.790261 13468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:36:45.791927 13468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:36:45.791981 13468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:36:45.792018 13468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:36:45.794155 13468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46731/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44937
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40997
--tserver_master_addrs=127.0.98.62:38263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:36:45.794418 13468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:36:45.794670 13468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:36:45.801041 13476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:45.801004 13474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:45.801657 13473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:45.802059 13468 server_base.cc:1034] running on GCE node
I20260928 02:36:45.802275 13468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:36:45.802548 13468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:36:45.803721 13468 hybrid_clock.cc:648] HybridClock initialized: now 1790563005803697 us; error 39 us; skew 500 ppm
Sep 28 02:36:45 dist-test-slave-fz47 krb5kdc[12137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:36:45.806234 13468 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 02:36:45.806726 13468 webserver.cc:460] Webserver started at http://127.0.98.3:40997/ using document root <none> and password file <none>
I20260928 02:36:45.806975 13468 fs_manager.cc:359] Metadata directory not provided
I20260928 02:36:45.807067 13468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:36:45 dist-test-slave-fz47 krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 02:36:45.822165 13468 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:36:45.823019 13484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:45.823508 13468 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:36:45.823599 13468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "9f063b3987a34ea499de06b3ef2e0fca"
format_stamp: "Formatted at 2026-09-28 02:36:45 on dist-test-slave-fz47"
server_key: "f1d527a5c1e5cfb7186b5035e70539fd"
server_key_iv: "825d2c4371cf31727a414e4f68963cc0"
server_key_version: "kuduclusterkey@0"
I20260928 02:36:45.824046 13468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:36:45.872123 13468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:36:45.874387 13468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:36:45.874538 13468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:36:45.874789 13468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:36:45.875160 13468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:36:45.875209 13468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:45.875238 13468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:36:45.875262 13468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:45.882300 13468 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44937
I20260928 02:36:45.882401 13598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44937 every 8 connection(s)
I20260928 02:36:45.882675 13468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
I20260928 02:36:45.888010 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 13468
Sep 28 02:36:45 dist-test-slave-fz47 krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:36:45.891690 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:45.883351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55779 (local address 127.0.98.62:38263)
0928 02:36:45.883409 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:45.883411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:45.883596 (+ 185us) server_negotiation.cc:407] Connection header received
0928 02:36:45.884084 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:45.884085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:45.884096 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:45.884166 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:45.884643 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:45.885437 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:45.886467 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:45.886673 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:45.886729 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:45.888821 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:45.888822 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:45.888823 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:45.888829 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:45.890288 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:45.890700 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:45.890702 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:45.890702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:45.890762 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:45.891020 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:45.891021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:45.891021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:45.891136 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:45.891189 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:45.891522 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 02:36:45.891600 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 02:36:45.892161 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:45.883321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38263 (local address 127.0.98.3:55779)
0928 02:36:45.883485 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:45.883525 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:45.883999 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:45.884223 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:45.884226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:45.884395 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:36:45.884579 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:45.884584 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:45.885511 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:45.885512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:45.886381 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:45.886384 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:45.886753 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:45.886754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:45.886867 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:45.887295 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:36:45.887303 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:36:45.888761 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:36:45.890380 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:45.890386 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:45.890388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:45.890639 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:45.890812 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:45.890812 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:45.890813 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:45.890929 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:45.891190 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:36:45.891196 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:36:45.891362 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 02:36:45.891783 (+ 421us) client_negotiation.cc:240] Negotiation successful
0928 02:36:45.891945 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":85,"threads_started":1}
I20260928 02:36:45.892580 13599 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38263
I20260928 02:36:45.892676 13599 heartbeater.cc:464] Registering TS with master...
I20260928 02:36:45.892855 13599 heartbeater.cc:511] Master 127.0.98.62:38263 requested a full tablet report, sending...
W20260928 02:36:45.893473 13599 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38263 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:36:45.893536 13599 heartbeater.cc:464] Registering TS with master...
I20260928 02:36:45.893597 13599 heartbeater.cc:511] Master 127.0.98.62:38263 requested a full tablet report, sending...
I20260928 02:36:45.893875 13599 heartbeater.cc:464] Registering TS with master...
I20260928 02:36:45.893944 13599 heartbeater.cc:511] Master 127.0.98.62:38263 requested a full tablet report, sending...
W20260928 02:36:45.894279 13599 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:36:45.898111 13602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:45.888370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51406 (local address 127.0.98.62:38263)
0928 02:36:45.888655 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:45.888656 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:45.888663 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:36:45.888684 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:45.888685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:45.888695 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:45.888759 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:45.889710 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:45.890458 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:45.891703 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:45.892114 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:45.892175 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:45.893874 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:45.893875 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:45.893876 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:45.893881 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:45.895447 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:45.895846 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:45.895847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:45.895848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:45.895909 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:45.896446 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:45.896447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:45.896447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:45.897358 (+ 911us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:45.897531 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:45.897919 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 02:36:45.898017 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":60,"threads_started":1}
Sep 28 02:36:46 dist-test-slave-fz47 krb5kdc[12137](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790563006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:36:46.524708 13331 heartbeater.cc:464] Registering TS with master...
I20260928 02:36:46.524863 13331 heartbeater.cc:511] Master 127.0.98.62:38263 requested a full tablet report, sending...
I20260928 02:36:46.525216 13133 ts_manager.cc:194] Registered new tserver with Master: 0676e01e15d245c6878a6d3d093d43da (127.0.98.1:39979)
I20260928 02:36:46.526224 13133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42017
I20260928 02:36:46.683890 13465 heartbeater.cc:464] Registering TS with master...
I20260928 02:36:46.684041 13465 heartbeater.cc:511] Master 127.0.98.62:38263 requested a full tablet report, sending...
I20260928 02:36:46.684304 13133 ts_manager.cc:194] Registered new tserver with Master: c12d1288231445a8844827cd2e10eca5 (127.0.98.2:41537)
I20260928 02:36:46.684844 13133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39507
Sep 28 02:36:46 dist-test-slave-fz47 krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 02:36:46.894677 13599 heartbeater.cc:464] Registering TS with master...
I20260928 02:36:46.894831 13599 heartbeater.cc:511] Master 127.0.98.62:38263 requested a full tablet report, sending...
I20260928 02:36:46.895153 13133 ts_manager.cc:194] Registered new tserver with Master: 9f063b3987a34ea499de06b3ef2e0fca (127.0.98.3:44937)
I20260928 02:36:46.895649 13133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55779
I20260928 02:36:46.900954 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:36:46.909633 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:46.901924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51418 (local address 127.0.98.62:38263)
0928 02:36:46.902206 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:46.902208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:46.902267 (+ 59us) server_negotiation.cc:407] Connection header received
0928 02:36:46.902342 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:46.902343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:46.902354 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:46.902415 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:46.903077 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:46.903854 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:46.904906 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:46.905222 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:46.905315 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:46.906649 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:46.906650 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:46.906651 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:46.906658 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:46.908371 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:46.908743 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:46.908744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:46.908744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:46.908805 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:46.908987 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:46.908987 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:46.908988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:46.909114 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:46.909207 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:46.909348 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 02:36:46.909431 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":71,"threads_started":1}
I20260928 02:36:46.911681 13133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:36:46.914047 13133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:36:46.924651 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:46.919987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47390 (local address 127.0.98.2:41537)
0928 02:36:46.920130 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:46.920135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:46.920334 (+ 199us) server_negotiation.cc:407] Connection header received
0928 02:36:46.920490 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:46.920493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:46.920509 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:46.920558 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:36:46.920821 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:46.921929 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:46.923418 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:46.924136 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:46.924186 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:46.924262 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:46.924530 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 02:36:46.924570 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":54,"threads_started":1}
I20260928 02:36:46.925163 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:46.919880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41537 (local address 127.0.0.1:47390)
0928 02:36:46.920224 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:46.920265 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:46.920374 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:46.920664 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:46.920667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:46.920669 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:36:46.920732 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:46.920734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:46.922026 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:46.922027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:46.923303 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:46.923307 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:46.924325 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:46.924326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:46.924415 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:46.924429 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 02:36:46.925011 (+ 582us) client_negotiation.cc:240] Negotiation successful
0928 02:36:46.925073 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":81,"threads_started":1}
I20260928 02:36:46.926139 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:46.920086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44937 (local address 127.0.0.1:56080)
0928 02:36:46.920916 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:46.920925 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:46.920962 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:46.921661 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:46.921662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:46.921664 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:36:46.921702 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:46.921703 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:46.923463 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:46.923464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:46.924494 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:46.924496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:46.925949 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:46.925950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:46.926012 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:46.926016 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:36:46.926043 (+ 27us) client_negotiation.cc:240] Negotiation successful
0928 02:36:46.926077 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"thread_start_us":50,"threads_started":1}
I20260928 02:36:46.926061 13387 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe44eded2bb491a82547412ca748766 (DEFAULT_TABLE table=test-table [id=2b677e039e9244a784210ad09bbbe0f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:36:46.926400 13387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe44eded2bb491a82547412ca748766. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:36:46.926683 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:46.920423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56080 (local address 127.0.98.3:44937)
0928 02:36:46.921479 (+ 1056us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:46.921484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:46.921492 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:36:46.921521 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:46.921523 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:46.921537 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:46.921599 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:36:46.921770 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:46.922746 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:46.924585 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:46.925252 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:46.926455 (+ 1203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:46.926543 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:46.926569 (+ 26us) server_negotiation.cc:299] Negotiation successful
0928 02:36:46.926609 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1009,"thread_start_us":49,"threads_started":1}
I20260928 02:36:46.927260 13533 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe44eded2bb491a82547412ca748766 (DEFAULT_TABLE table=test-table [id=2b677e039e9244a784210ad09bbbe0f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:36:46.927562 13533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe44eded2bb491a82547412ca748766. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:36:46.928704 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:46.920199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39979 (local address 127.0.0.1:42434)
0928 02:36:46.921339 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:46.921347 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:46.921385 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:46.923348 (+ 1963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:46.923349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:46.923351 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:36:46.923388 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:46.923389 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:46.926510 (+ 3121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:46.926511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:46.927499 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:46.927502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:46.928503 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:46.928504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:46.928560 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:46.928564 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:36:46.928609 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 02:36:46.928642 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1127,"thread_start_us":45,"threads_started":1}
I20260928 02:36:46.928920 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:46.920559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42434 (local address 127.0.98.1:39979)
0928 02:36:46.922878 (+ 2319us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:46.923133 (+ 255us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:46.923143 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:36:46.923175 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:46.923177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:46.923191 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:46.923273 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:36:46.925348 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:46.926425 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:46.927602 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:46.928335 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:46.928401 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:46.928469 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:46.928807 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 02:36:46.928848 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2285,"thread_start_us":46,"threads_started":1}
I20260928 02:36:46.931993 13255 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe44eded2bb491a82547412ca748766 (DEFAULT_TABLE table=test-table [id=2b677e039e9244a784210ad09bbbe0f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:36:46.932168 13633 tablet_bootstrap.cc:492] T 3fe44eded2bb491a82547412ca748766 P 9f063b3987a34ea499de06b3ef2e0fca: Bootstrap starting.
I20260928 02:36:46.932312 13255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe44eded2bb491a82547412ca748766. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:36:46.932768 13633 tablet_bootstrap.cc:654] T 3fe44eded2bb491a82547412ca748766 P 9f063b3987a34ea499de06b3ef2e0fca: Neither blocks nor log segments found. Creating new log.
I20260928 02:36:46.933049 13633 log.cc:824] T 3fe44eded2bb491a82547412ca748766 P 9f063b3987a34ea499de06b3ef2e0fca: Log is configured to *not* fsync() on all Append() calls
I20260928 02:36:46.933368 13632 tablet_bootstrap.cc:492] T 3fe44eded2bb491a82547412ca748766 P c12d1288231445a8844827cd2e10eca5: Bootstrap starting.
I20260928 02:36:46.933949 13632 tablet_bootstrap.cc:654] T 3fe44eded2bb491a82547412ca748766 P c12d1288231445a8844827cd2e10eca5: Neither blocks nor log segments found. Creating new log.
I20260928 02:36:46.934206 13632 log.cc:824] T 3fe44eded2bb491a82547412ca748766 P c12d1288231445a8844827cd2e10eca5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:36:46.934919 13633 tablet_bootstrap.cc:492] T 3fe44eded2bb491a82547412ca748766 P 9f063b3987a34ea499de06b3ef2e0fca: No bootstrap required, opened a new log
I20260928 02:36:46.934978 13633 ts_tablet_manager.cc:1397] T 3fe44eded2bb491a82547412ca748766 P 9f063b3987a34ea499de06b3ef2e0fca: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 02:36:46.937445 13633 raft_consensus.cc:348] T 3fe44eded2bb491a82547412ca748766 P 9f063b3987a34ea499de06b3ef2e0fca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d1288231445a8844827cd2e10eca5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41537 } } peers { permanent_uuid: "9f063b3987a34ea499de06b3ef2e0fca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44937 } } peers { permanent_uuid: "0676e01e15d245c6878a6d3d093d43da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39979 } }
I20260928 02:36:46.937625 13636 tablet_bootstrap.cc:492] T 3fe44eded2bb491a82547412ca748766 P 0676e01e15d245c6878a6d3d093d43da: Bootstrap starting.
I20260928 02:36:46.937700 13633 raft_consensus.cc:374] T 3fe44eded2bb491a82547412ca748766 P 9f063b3987a34ea499de06b3ef2e0fca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:36:46.937798 13633 raft_consensus.cc:729] T 3fe44eded2bb491a82547412ca748766 P 9f063b3987a34ea499de06b3ef2e0fca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f063b3987a34ea499de06b3ef2e0fca, State: Initialized, Role: FOLLOWER
I20260928 02:36:46.937988 13633 consensus_queue.cc:260] T 3fe44eded2bb491a82547412ca748766 P 9f063b3987a34ea499de06b3ef2e0fca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d1288231445a8844827cd2e10eca5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41537 } } peers { permanent_uuid: "9f063b3987a34ea499de06b3ef2e0fca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44937 } } peers { permanent_uuid: "0676e01e15d245c6878a6d3d093d43da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39979 } }
I20260928 02:36:46.938211 13636 tablet_bootstrap.cc:654] T 3fe44eded2bb491a82547412ca748766 P 0676e01e15d245c6878a6d3d093d43da: Neither blocks nor log segments found. Creating new log.
I20260928 02:36:46.938357 13633 ts_tablet_manager.cc:1428] T 3fe44eded2bb491a82547412ca748766 P 9f063b3987a34ea499de06b3ef2e0fca: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:36:46.938527 13599 heartbeater.cc:503] Master 127.0.98.62:38263 was elected leader, sending a full tablet report...
I20260928 02:36:46.938576 13632 tablet_bootstrap.cc:492] T 3fe44eded2bb491a82547412ca748766 P c12d1288231445a8844827cd2e10eca5: No bootstrap required, opened a new log
I20260928 02:36:46.938627 13632 ts_tablet_manager.cc:1397] T 3fe44eded2bb491a82547412ca748766 P c12d1288231445a8844827cd2e10eca5: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 02:36:46.940582 13636 log.cc:824] T 3fe44eded2bb491a82547412ca748766 P 0676e01e15d245c6878a6d3d093d43da: Log is configured to *not* fsync() on all Append() calls
I20260928 02:36:46.941313 13636 tablet_bootstrap.cc:492] T 3fe44eded2bb491a82547412ca748766 P 0676e01e15d245c6878a6d3d093d43da: No bootstrap required, opened a new log
I20260928 02:36:46.941385 13636 ts_tablet_manager.cc:1397] T 3fe44eded2bb491a82547412ca748766 P 0676e01e15d245c6878a6d3d093d43da: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 02:36:46.941457 13632 raft_consensus.cc:348] T 3fe44eded2bb491a82547412ca748766 P c12d1288231445a8844827cd2e10eca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d1288231445a8844827cd2e10eca5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41537 } } peers { permanent_uuid: "9f063b3987a34ea499de06b3ef2e0fca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44937 } } peers { permanent_uuid: "0676e01e15d245c6878a6d3d093d43da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39979 } }
I20260928 02:36:46.941593 13632 raft_consensus.cc:374] T 3fe44eded2bb491a82547412ca748766 P c12d1288231445a8844827cd2e10eca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:36:46.941618 13632 raft_consensus.cc:729] T 3fe44eded2bb491a82547412ca748766 P c12d1288231445a8844827cd2e10eca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c12d1288231445a8844827cd2e10eca5, State: Initialized, Role: FOLLOWER
I20260928 02:36:46.941735 13632 consensus_queue.cc:260] T 3fe44eded2bb491a82547412ca748766 P c12d1288231445a8844827cd2e10eca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d1288231445a8844827cd2e10eca5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41537 } } peers { permanent_uuid: "9f063b3987a34ea499de06b3ef2e0fca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44937 } } peers { permanent_uuid: "0676e01e15d245c6878a6d3d093d43da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39979 } }
I20260928 02:36:46.941939 13632 ts_tablet_manager.cc:1428] T 3fe44eded2bb491a82547412ca748766 P c12d1288231445a8844827cd2e10eca5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:36:46.942027 13465 heartbeater.cc:503] Master 127.0.98.62:38263 was elected leader, sending a full tablet report...
I20260928 02:36:46.943764 13636 raft_consensus.cc:348] T 3fe44eded2bb491a82547412ca748766 P 0676e01e15d245c6878a6d3d093d43da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d1288231445a8844827cd2e10eca5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41537 } } peers { permanent_uuid: "9f063b3987a34ea499de06b3ef2e0fca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44937 } } peers { permanent_uuid: "0676e01e15d245c6878a6d3d093d43da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39979 } }
I20260928 02:36:46.943907 13636 raft_consensus.cc:374] T 3fe44eded2bb491a82547412ca748766 P 0676e01e15d245c6878a6d3d093d43da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:36:46.943941 13636 raft_consensus.cc:729] T 3fe44eded2bb491a82547412ca748766 P 0676e01e15d245c6878a6d3d093d43da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0676e01e15d245c6878a6d3d093d43da, State: Initialized, Role: FOLLOWER
I20260928 02:36:46.944041 13636 consensus_queue.cc:260] T 3fe44eded2bb491a82547412ca748766 P 0676e01e15d245c6878a6d3d093d43da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d1288231445a8844827cd2e10eca5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41537 } } peers { permanent_uuid: "9f063b3987a34ea499de06b3ef2e0fca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44937 } } peers { permanent_uuid: "0676e01e15d245c6878a6d3d093d43da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39979 } }
I20260928 02:36:46.944263 13636 ts_tablet_manager.cc:1428] T 3fe44eded2bb491a82547412ca748766 P 0676e01e15d245c6878a6d3d093d43da: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:36:46.944572 13331 heartbeater.cc:503] Master 127.0.98.62:38263 was elected leader, sending a full tablet report...
Sep 28 02:36:46 dist-test-slave-fz47 krb5kdc[12137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260928 02:36:47.006910 13332 tablet.cc:2369] T 3fe44eded2bb491a82547412ca748766 P 0676e01e15d245c6878a6d3d093d43da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:36:47.111306 13640 raft_consensus.cc:482] T 3fe44eded2bb491a82547412ca748766 P 0676e01e15d245c6878a6d3d093d43da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:36:47.111474 13640 raft_consensus.cc:504] T 3fe44eded2bb491a82547412ca748766 P 0676e01e15d245c6878a6d3d093d43da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d1288231445a8844827cd2e10eca5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41537 } } peers { permanent_uuid: "9f063b3987a34ea499de06b3ef2e0fca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44937 } } peers { permanent_uuid: "0676e01e15d245c6878a6d3d093d43da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39979 } }
I20260928 02:36:47.111876 13640 leader_election.cc:288] T 3fe44eded2bb491a82547412ca748766 P 0676e01e15d245c6878a6d3d093d43da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c12d1288231445a8844827cd2e10eca5 (127.0.98.2:41537), 9f063b3987a34ea499de06b3ef2e0fca (127.0.98.3:44937)
I20260928 02:36:47.119582 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:47.112741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52569 (local address 127.0.98.2:41537)
0928 02:36:47.112838 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:47.112840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:47.114584 (+ 1744us) server_negotiation.cc:407] Connection header received
0928 02:36:47.114911 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:47.114913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:47.114929 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:47.114959 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:36:47.115201 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:47.115934 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:47.118433 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:47.119008 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:47.119057 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:47.119069 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:47.119266 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 02:36:47.119306 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:36:47.120033 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:47.112183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41537 (local address 127.0.98.1:52569)
0928 02:36:47.114497 (+ 2314us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:47.114511 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:47.114618 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:47.115030 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:47.115031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:47.115033 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:36:47.115127 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:47.115129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:47.116038 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:47.116039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:47.118291 (+ 2252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:47.118294 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:47.119124 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:47.119125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:47.119197 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:47.119200 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:36:47.119893 (+ 693us) client_negotiation.cc:240] Negotiation successful
0928 02:36:47.119937 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2292,"thread_start_us":89,"threads_started":1}
I20260928 02:36:47.120548 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:47.112790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41701 (local address 127.0.98.3:44937)
0928 02:36:47.112871 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:47.112873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:47.117408 (+ 4535us) server_negotiation.cc:407] Connection header received
0928 02:36:47.117510 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:47.117512 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:47.117523 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:47.117551 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:36:47.117751 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:47.118430 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:47.119626 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:47.120180 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:47.120223 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:47.120234 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:47.120443 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 02:36:47.120484 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:36:47.120494 13417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe44eded2bb491a82547412ca748766" candidate_uuid: "0676e01e15d245c6878a6d3d093d43da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d1288231445a8844827cd2e10eca5" is_pre_election: true
I20260928 02:36:47.120673 13417 raft_consensus.cc:2454] T 3fe44eded2bb491a82547412ca748766 P c12d1288231445a8844827cd2e10eca5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0676e01e15d245c6878a6d3d093d43da in term 0.
I20260928 02:36:47.120924 13219 leader_election.cc:302] T 3fe44eded2bb491a82547412ca748766 P 0676e01e15d245c6878a6d3d093d43da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0676e01e15d245c6878a6d3d093d43da, c12d1288231445a8844827cd2e10eca5; no voters:
I20260928 02:36:47.121085 13640 raft_consensus.cc:2781] T 3fe44eded2bb491a82547412ca748766 P 0676e01e15d245c6878a6d3d093d43da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:36:47.121181 13640 raft_consensus.cc:482] T 3fe44eded2bb491a82547412ca748766 P 0676e01e15d245c6878a6d3d093d43da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:36:47.121203 13640 raft_consensus.cc:3037] T 3fe44eded2bb491a82547412ca748766 P 0676e01e15d245c6878a6d3d093d43da [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:36:47.122099 13640 raft_consensus.cc:504] T 3fe44eded2bb491a82547412ca748766 P 0676e01e15d245c6878a6d3d093d43da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d1288231445a8844827cd2e10eca5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41537 } } peers { permanent_uuid: "9f063b3987a34ea499de06b3ef2e0fca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44937 } } peers { permanent_uuid: "0676e01e15d245c6878a6d3d093d43da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39979 } }
I20260928 02:36:47.122225 13640 leader_election.cc:288] T 3fe44eded2bb491a82547412ca748766 P 0676e01e15d245c6878a6d3d093d43da [CANDIDATE]: Term 1 election: Requested vote from peers c12d1288231445a8844827cd2e10eca5 (127.0.98.2:41537), 9f063b3987a34ea499de06b3ef2e0fca (127.0.98.3:44937)
I20260928 02:36:47.122520 13417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe44eded2bb491a82547412ca748766" candidate_uuid: "0676e01e15d245c6878a6d3d093d43da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c12d1288231445a8844827cd2e10eca5"
I20260928 02:36:47.122599 13417 raft_consensus.cc:3037] T 3fe44eded2bb491a82547412ca748766 P c12d1288231445a8844827cd2e10eca5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:36:47.123479 13417 raft_consensus.cc:2454] T 3fe44eded2bb491a82547412ca748766 P c12d1288231445a8844827cd2e10eca5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0676e01e15d245c6878a6d3d093d43da in term 1.
I20260928 02:36:47.123803 13219 leader_election.cc:302] T 3fe44eded2bb491a82547412ca748766 P 0676e01e15d245c6878a6d3d093d43da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0676e01e15d245c6878a6d3d093d43da, c12d1288231445a8844827cd2e10eca5; no voters:
I20260928 02:36:47.123937 13640 raft_consensus.cc:2781] T 3fe44eded2bb491a82547412ca748766 P 0676e01e15d245c6878a6d3d093d43da [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:36:47.124087 13640 raft_consensus.cc:686] T 3fe44eded2bb491a82547412ca748766 P 0676e01e15d245c6878a6d3d093d43da [term 1 LEADER]: Becoming Leader. State: Replica: 0676e01e15d245c6878a6d3d093d43da, State: Running, Role: LEADER
I20260928 02:36:47.124177 13640 consensus_queue.cc:237] T 3fe44eded2bb491a82547412ca748766 P 0676e01e15d245c6878a6d3d093d43da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d1288231445a8844827cd2e10eca5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41537 } } peers { permanent_uuid: "9f063b3987a34ea499de06b3ef2e0fca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44937 } } peers { permanent_uuid: "0676e01e15d245c6878a6d3d093d43da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39979 } }
I20260928 02:36:47.125005 13133 catalog_manager.cc:5495] T 3fe44eded2bb491a82547412ca748766 P 0676e01e15d245c6878a6d3d093d43da reported cstate change: term changed from 0 to 1, leader changed from <none> to 0676e01e15d245c6878a6d3d093d43da (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0676e01e15d245c6878a6d3d093d43da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c12d1288231445a8844827cd2e10eca5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f063b3987a34ea499de06b3ef2e0fca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0676e01e15d245c6878a6d3d093d43da" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39979 } health_report { overall_health: HEALTHY } } }
I20260928 02:36:47.121006 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:47.112393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44937 (local address 127.0.98.1:41701)
0928 02:36:47.117331 (+ 4938us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:47.117344 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:47.117436 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:47.117610 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:47.117611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:47.117613 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:36:47.117677 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:47.117679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:47.118511 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:47.118512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:47.119540 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:47.119542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:47.120305 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:47.120305 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:47.120366 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:47.120368 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:36:47.120823 (+ 455us) client_negotiation.cc:240] Negotiation successful
0928 02:36:47.120877 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4920,"thread_start_us":83,"threads_started":1}
I20260928 02:36:47.126606 13553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe44eded2bb491a82547412ca748766" candidate_uuid: "0676e01e15d245c6878a6d3d093d43da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f063b3987a34ea499de06b3ef2e0fca" is_pre_election: true
I20260928 02:36:47.126750 13553 raft_consensus.cc:2454] T 3fe44eded2bb491a82547412ca748766 P 9f063b3987a34ea499de06b3ef2e0fca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0676e01e15d245c6878a6d3d093d43da in term 0.
I20260928 02:36:47.126848 13552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe44eded2bb491a82547412ca748766" candidate_uuid: "0676e01e15d245c6878a6d3d093d43da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f063b3987a34ea499de06b3ef2e0fca"
I20260928 02:36:47.126909 13552 raft_consensus.cc:3037] T 3fe44eded2bb491a82547412ca748766 P 9f063b3987a34ea499de06b3ef2e0fca [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:36:47.127736 13552 raft_consensus.cc:2454] T 3fe44eded2bb491a82547412ca748766 P 9f063b3987a34ea499de06b3ef2e0fca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0676e01e15d245c6878a6d3d093d43da in term 1.
W20260928 02:36:47.133908 13600 tablet.cc:2369] T 3fe44eded2bb491a82547412ca748766 P 9f063b3987a34ea499de06b3ef2e0fca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:36:47.149703 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:47.146417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42450 (local address 127.0.98.1:39979)
0928 02:36:47.146459 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:47.146461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:47.146468 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:36:47.146508 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:47.146509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:47.146530 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:47.146592 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:36:47.146778 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:47.147500 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:47.148609 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:47.148915 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:47.148973 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:47.149164 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:36:47.149315 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:36:47.149390 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:47.149605 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 02:36:47.149625 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:36:47.154385 13552 raft_consensus.cc:1261] T 3fe44eded2bb491a82547412ca748766 P 9f063b3987a34ea499de06b3ef2e0fca [term 1 FOLLOWER]: Refusing update from remote peer 0676e01e15d245c6878a6d3d093d43da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:36:47.154820 13645 consensus_queue.cc:1035] T 3fe44eded2bb491a82547412ca748766 P 0676e01e15d245c6878a6d3d093d43da [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f063b3987a34ea499de06b3ef2e0fca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:36:47.157033 13417 raft_consensus.cc:1261] T 3fe44eded2bb491a82547412ca748766 P c12d1288231445a8844827cd2e10eca5 [term 1 FOLLOWER]: Refusing update from remote peer 0676e01e15d245c6878a6d3d093d43da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:36:47.157387 13645 consensus_queue.cc:1035] T 3fe44eded2bb491a82547412ca748766 P 0676e01e15d245c6878a6d3d093d43da [LEADER]: Connected to new peer: Peer: permanent_uuid: "c12d1288231445a8844827cd2e10eca5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:36:47.161046 13648 mvcc.cc:205] Tried to move back new op lower bound from 7334146077301235712 to 7334146077181218816. Current Snapshot: MvccSnapshot[applied={T|T < 7334146077301235712}]
I20260928 02:36:47.161659 13651 mvcc.cc:205] Tried to move back new op lower bound from 7334146077301235712 to 7334146077181218816. Current Snapshot: MvccSnapshot[applied={T|T < 7334146077301235712}]
I20260928 02:36:47.165892 13649 mvcc.cc:205] Tried to move back new op lower bound from 7334146077301235712 to 7334146077181218816. Current Snapshot: MvccSnapshot[applied={T|T < 7334146077301235712}]
I20260928 02:36:47.166074 13133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:36:47.166160 13133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:36:47.166930 13133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7c2f9d26ec541b4841a29fbe01d8841: Sending DeleteTablet for 3 replicas of tablet 3fe44eded2bb491a82547412ca748766
I20260928 02:36:47.167402 13387 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fe44eded2bb491a82547412ca748766 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:36:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47390
I20260928 02:36:47.167606 13533 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fe44eded2bb491a82547412ca748766 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:36:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56080
I20260928 02:36:47.167791 13255 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fe44eded2bb491a82547412ca748766 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:36:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42434
I20260928 02:36:47.168416 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 13171
I20260928 02:36:47.172276 13657 tablet_replica.cc:330] T 3fe44eded2bb491a82547412ca748766 P c12d1288231445a8844827cd2e10eca5: stopping tablet replica
I20260928 02:36:47.172425 13658 tablet_replica.cc:330] T 3fe44eded2bb491a82547412ca748766 P 9f063b3987a34ea499de06b3ef2e0fca: stopping tablet replica
I20260928 02:36:47.172729 13658 raft_consensus.cc:2229] T 3fe44eded2bb491a82547412ca748766 P 9f063b3987a34ea499de06b3ef2e0fca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:36:47.173226 13657 raft_consensus.cc:2229] T 3fe44eded2bb491a82547412ca748766 P c12d1288231445a8844827cd2e10eca5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:36:47.173435 13658 raft_consensus.cc:2258] T 3fe44eded2bb491a82547412ca748766 P 9f063b3987a34ea499de06b3ef2e0fca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:36:47.173547 13657 raft_consensus.cc:2258] T 3fe44eded2bb491a82547412ca748766 P c12d1288231445a8844827cd2e10eca5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:36:47.174189 13657 ts_tablet_manager.cc:1905] T 3fe44eded2bb491a82547412ca748766 P c12d1288231445a8844827cd2e10eca5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:36:47.174556 13658 ts_tablet_manager.cc:1905] T 3fe44eded2bb491a82547412ca748766 P 9f063b3987a34ea499de06b3ef2e0fca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:36:47.177516 13657 ts_tablet_manager.cc:1918] T 3fe44eded2bb491a82547412ca748766 P c12d1288231445a8844827cd2e10eca5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:36:47.177667 13658 ts_tablet_manager.cc:1918] T 3fe44eded2bb491a82547412ca748766 P 9f063b3987a34ea499de06b3ef2e0fca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:36:47.177772 13657 log.cc:1196] T 3fe44eded2bb491a82547412ca748766 P c12d1288231445a8844827cd2e10eca5: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-1/wal/wals/3fe44eded2bb491a82547412ca748766
I20260928 02:36:47.177881 13658 log.cc:1196] T 3fe44eded2bb491a82547412ca748766 P 9f063b3987a34ea499de06b3ef2e0fca: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790562862804661-392-0/minicluster-data/ts-2/wal/wals/3fe44eded2bb491a82547412ca748766
I20260928 02:36:47.178177 13658 ts_tablet_manager.cc:1939] T 3fe44eded2bb491a82547412ca748766 P 9f063b3987a34ea499de06b3ef2e0fca: Deleting consensus metadata
I20260928 02:36:47.178444 13657 ts_tablet_manager.cc:1939] T 3fe44eded2bb491a82547412ca748766 P c12d1288231445a8844827cd2e10eca5: Deleting consensus metadata
I20260928 02:36:47.182411 13118 catalog_manager.cc:4841] TS c12d1288231445a8844827cd2e10eca5 (127.0.98.2:41537): tablet 3fe44eded2bb491a82547412ca748766 (table test-table [id=2b677e039e9244a784210ad09bbbe0f4]) successfully deleted
I20260928 02:36:47.182629 13118 catalog_manager.cc:4841] TS 9f063b3987a34ea499de06b3ef2e0fca (127.0.98.3:44937): tablet 3fe44eded2bb491a82547412ca748766 (table test-table [id=2b677e039e9244a784210ad09bbbe0f4]) successfully deleted
W20260928 02:36:47.182874 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39979 (error 108)
I20260928 02:36:47.183089 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 13333
I20260928 02:36:47.189227 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:47.189112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39979 (local address 127.0.0.1:42458)
0928 02:36:47.189159 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:47.189189 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 02:36:47.189462 13118 catalog_manager.cc:4584] TS 0676e01e15d245c6878a6d3d093d43da (127.0.98.1:39979): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3fe44eded2bb491a82547412ca748766: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39979: connect: Connection refused (error 111)
I20260928 02:36:47.189692 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 13468
I20260928 02:36:47.194700 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 13100
I20260928 02:36:47.200915 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 02:36:47.534704 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 02:36:47.534798 392 mini_ranger.cc:66] Stopping Ranger...
I20260928 02:36:47.868561 392 mini_ranger.cc:68] Stopped Ranger
2026-09-28 02:36:47.868 UTC [12167] LOG: received smart shutdown request
2026-09-28 02:36:47.871 UTC [12167] LOG: background worker "logical replication launcher" (PID 12176) exited with exit code 1
2026-09-28 02:36:47.871 UTC [12171] LOG: shutting down
2026-09-28 02:36:48.008 UTC [12167] LOG: database system is shut down
2026-09-28T02:36:48Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (45162 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:36:48 dist-test-slave-fz47 krb5kdc[13669](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:36:48 dist-test-slave-fz47 krb5kdc[13669](info): set up 2 sockets
Sep 28 02:36:48 dist-test-slave-fz47 krb5kdc[13669](info): commencing operation
krb5kdc: starting...
W20260928 02:36:49.028831 392 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:36:49 dist-test-slave-fz47 krb5kdc[13669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:36:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:36:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 02:36:49.137585 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37939
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37939,127.0.98.61:40077,127.0.98.60:42121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:36:49.215652 13682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:36:49.215859 13682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:36:49.215890 13682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:36:49.217012 13682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:36:49.217051 13682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:36:49.217070 13682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:36:49.217087 13682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:36:49.217101 13682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:36:49.218647 13682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37939,127.0.98.61:40077,127.0.98.60:42121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:36:49.218885 13682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:36:49.219079 13682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:36:49.223632 13687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:49.223713 13682 server_base.cc:1034] running on GCE node
W20260928 02:36:49.223644 13688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:49.223632 13690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:49.224086 13682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:36:49.224371 13682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:36:49.225576 13682 hybrid_clock.cc:648] HybridClock initialized: now 1790563009225563 us; error 31 us; skew 500 ppm
Sep 28 02:36:49 dist-test-slave-fz47 krb5kdc[13669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:36:49.227931 13682 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:36:49.228348 13682 webserver.cc:460] Webserver started at http://127.0.98.62:44827/ using document root <none> and password file <none>
I20260928 02:36:49.228513 13682 fs_manager.cc:359] Metadata directory not provided
I20260928 02:36:49.228549 13682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:36:49.228634 13682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:36:49.229647 13682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "f125a383e1b64933869af0e14b436509"
format_stamp: "Formatted at 2026-09-28 02:36:49 on dist-test-slave-fz47"
server_key: "54459ec8de274cdec3570c6cb6a13e71"
server_key_iv: "9abe4b29e84b6dacbe55a0ec678dd89d"
server_key_version: "encryptionkey@0"
I20260928 02:36:49.229938 13682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "f125a383e1b64933869af0e14b436509"
format_stamp: "Formatted at 2026-09-28 02:36:49 on dist-test-slave-fz47"
server_key: "54459ec8de274cdec3570c6cb6a13e71"
server_key_iv: "9abe4b29e84b6dacbe55a0ec678dd89d"
server_key_version: "encryptionkey@0"
I20260928 02:36:49.231076 13682 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:36:49.231671 13697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:49.231824 13682 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:36:49.231902 13682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "f125a383e1b64933869af0e14b436509"
format_stamp: "Formatted at 2026-09-28 02:36:49 on dist-test-slave-fz47"
server_key: "54459ec8de274cdec3570c6cb6a13e71"
server_key_iv: "9abe4b29e84b6dacbe55a0ec678dd89d"
server_key_version: "encryptionkey@0"
I20260928 02:36:49.231956 13682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:36:49.252557 13682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:36:49.254722 13682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:36:49.254842 13682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:36:49.258421 13682 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37939
I20260928 02:36:49.258468 13749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37939 every 8 connection(s)
I20260928 02:36:49.258774 13682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:36:49.259603 13750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:36:49.261171 13750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:36:49.261678 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 13682
I20260928 02:36:49.261786 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:36:49.261983 392 external_mini_cluster.cc:1428] Setting key 7e6fb4e2f40d66f4e97d26469c8b145b
Sep 28 02:36:49 dist-test-slave-fz47 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260928 02:36:49.270308 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.261917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.0.1:55110)
0928 02:36:49.262084 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:49.262124 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:49.262630 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:49.262874 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:49.262877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:49.263150 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:36:49.263338 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:49.263344 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.264296 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.264297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:49.265007 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:49.265010 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.265514 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.265516 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:49.265584 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:49.266069 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:36:49.266077 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:36:49.267359 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:36:49.268894 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:49.268899 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:49.268901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:49.269128 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:49.269373 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:49.269373 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:49.269374 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:49.269456 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:49.269805 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:36:49.269809 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:36:49.269870 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 02:36:49.269955 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 02:36:49.270106 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
I20260928 02:36:49.270308 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.261918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55110 (local address 127.0.98.62:37939)
0928 02:36:49.262112 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:49.262122 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:49.262250 (+ 128us) server_negotiation.cc:407] Connection header received
0928 02:36:49.262769 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:49.262772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:49.262790 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:49.262865 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:49.263421 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.264214 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.265074 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.265445 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.265539 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:49.267433 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:49.267439 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:49.267440 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:49.267446 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:49.268822 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:49.269199 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:49.269202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:49.269202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:49.269317 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:49.269507 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:49.269508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:49.269508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:49.269744 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:49.269815 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:49.270037 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 02:36:49.270195 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":159,"spinlock_wait_cycles":6912,"thread_start_us":103,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260928 02:36:49.271099 13750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40077 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:36:49.271401 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.271306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40077 (local address 127.0.0.1:45646)
0928 02:36:49.271346 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:49.271385 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 02:36:49.271519 13698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40077: connect: Connection refused (error 111)
I20260928 02:36:49.271924 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.271845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40077 (local address 127.0.0.1:45662)
0928 02:36:49.271880 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:49.271903 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 02:36:49.272013 13750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40077: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40077: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260928 02:36:49.283646 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40077
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37939,127.0.98.61:40077,127.0.98.60:42121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:36:49.321319 13750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40077 } attempt: 1
I20260928 02:36:49.321820 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.321684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40077 (local address 127.0.0.1:45672)
0928 02:36:49.321744 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:49.321789 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 02:36:49.322175 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.322079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40077 (local address 127.0.0.1:45686)
0928 02:36:49.322122 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:49.322151 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 02:36:49.322263 13750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40077: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40077: connect: Connection refused (error 111)
W20260928 02:36:49.361610 13757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:36:49.361806 13757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:36:49.361835 13757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:36:49.362929 13757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:36:49.362968 13757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:36:49.362985 13757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:36:49.363001 13757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:36:49.363016 13757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:36:49.364490 13757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37939,127.0.98.61:40077,127.0.98.60:42121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:36:49.364727 13757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:36:49.364920 13757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:36:49.369545 13763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:49.369585 13762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:49.369575 13765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:49.369624 13757 server_base.cc:1034] running on GCE node
I20260928 02:36:49.369952 13757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:36:49.370234 13757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:36:49.371387 13757 hybrid_clock.cc:648] HybridClock initialized: now 1790563009371372 us; error 29 us; skew 500 ppm
Sep 28 02:36:49 dist-test-slave-fz47 krb5kdc[13669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:36:49.373662 13757 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260928 02:36:49.374058 13757 webserver.cc:460] Webserver started at http://127.0.98.61:45493/ using document root <none> and password file <none>
I20260928 02:36:49.374250 13757 fs_manager.cc:359] Metadata directory not provided
I20260928 02:36:49.374287 13757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:36:49.374375 13757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:36:49.375442 13757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-1/data/instance:
uuid: "4fa5a7e6d51148c7ba73a85e581fcfb2"
format_stamp: "Formatted at 2026-09-28 02:36:49 on dist-test-slave-fz47"
server_key: "251582ef83ddc37ae9f4bf118699a299"
server_key_iv: "0b9e409d06d2056a7a000403643b04ae"
server_key_version: "encryptionkey@0"
I20260928 02:36:49.375834 13757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-1/wal/instance:
uuid: "4fa5a7e6d51148c7ba73a85e581fcfb2"
format_stamp: "Formatted at 2026-09-28 02:36:49 on dist-test-slave-fz47"
server_key: "251582ef83ddc37ae9f4bf118699a299"
server_key_iv: "0b9e409d06d2056a7a000403643b04ae"
server_key_version: "encryptionkey@0"
I20260928 02:36:49.377477 13757 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:36:49.378352 13772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:49.378533 13757 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:36:49.378626 13757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-1/wal
uuid: "4fa5a7e6d51148c7ba73a85e581fcfb2"
format_stamp: "Formatted at 2026-09-28 02:36:49 on dist-test-slave-fz47"
server_key: "251582ef83ddc37ae9f4bf118699a299"
server_key_iv: "0b9e409d06d2056a7a000403643b04ae"
server_key_version: "encryptionkey@0"
I20260928 02:36:49.378703 13757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:36:49.390390 13750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40077 } attempt: 2
I20260928 02:36:49.390765 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.390657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40077 (local address 127.0.0.1:45698)
0928 02:36:49.390705 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:49.390737 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 02:36:49.391044 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.390965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40077 (local address 127.0.0.1:45710)
0928 02:36:49.390998 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:49.391021 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 02:36:49.391114 13750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40077: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40077: connect: Connection refused (error 111)
I20260928 02:36:49.402482 13757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:36:49.404506 13757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:36:49.404649 13757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:36:49.408473 13757 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40077
I20260928 02:36:49.408512 13824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40077 every 8 connection(s)
I20260928 02:36:49.408828 13757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-1/data/info.pb
I20260928 02:36:49.409653 13825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:36:49.411140 13825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:36:49 dist-test-slave-fz47 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:36:49.418025 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 13757
I20260928 02:36:49.418124 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-1/wal/instance
I20260928 02:36:49.418304 392 external_mini_cluster.cc:1428] Setting key 0f3fa8c5a9f7e950c3de953bacb388b3
I20260928 02:36:49.419204 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.411800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55118 (local address 127.0.98.62:37939)
0928 02:36:49.411871 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:49.411873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:49.412019 (+ 146us) server_negotiation.cc:407] Connection header received
0928 02:36:49.412471 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:49.412472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:49.412484 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:49.412519 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:49.413084 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.413594 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.414543 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.414744 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.414812 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:49.416628 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:49.416629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:49.416629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:49.416636 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:49.417929 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:49.418355 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:49.418356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:49.418357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:49.418425 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:49.418646 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:49.418647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:49.418647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:49.418781 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:49.418841 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:49.419046 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 02:36:49.419119 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:36:49.419337 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.411761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.0.1:55118)
0928 02:36:49.411908 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:49.411945 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:49.412388 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:49.412579 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:49.412582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:49.412827 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:36:49.413024 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:49.413028 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.413655 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.413656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:49.414463 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:49.414465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.414815 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.414816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:49.414909 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:49.415310 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:36:49.415317 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:36:49.416541 (+ 1224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:36:49.417997 (+ 1456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:49.418003 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:49.418005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:49.418289 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:49.418493 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:49.418494 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:49.418494 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:49.418594 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:49.418831 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:36:49.418834 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:36:49.418946 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 02:36:49.419019 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:36:49.419131 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260928 02:36:49.419902 13825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40077 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:36:49 dist-test-slave-fz47 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260928 02:36:49.426282 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.420129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40077 (local address 127.0.0.1:45722)
0928 02:36:49.420175 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:49.420184 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:49.420215 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:49.420461 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:49.420462 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:49.420564 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:36:49.420612 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:49.420613 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.421474 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.421475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:49.421955 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:49.421956 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.422352 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.422353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:49.422411 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:49.422746 (+ 335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:36:49.422751 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:36:49.423879 (+ 1128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:36:49.425384 (+ 1505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:49.425385 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:49.425386 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:49.425526 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:49.425696 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:49.425697 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:49.425697 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:49.425748 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:49.425993 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:36:49.425993 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:36:49.426029 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 02:36:49.426087 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 02:36:49.426177 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 02:36:49.426362 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.420196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45722 (local address 127.0.98.61:40077)
0928 02:36:49.420327 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:49.420331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:49.420351 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:36:49.420368 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:49.420370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:49.420382 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:49.420443 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:49.420692 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.421404 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.422017 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.422295 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.422351 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:49.423949 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:49.423956 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:49.423957 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:49.423963 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:49.425314 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:49.425588 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:49.425591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:49.425591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:49.425640 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:49.425798 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:49.425799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:49.425799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:49.425938 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:49.426004 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:49.426113 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 02:36:49.426236 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
I20260928 02:36:49.426919 13825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42121 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260928 02:36:49.427199 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.427109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42121 (local address 127.0.0.1:46344)
0928 02:36:49.427141 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:49.427181 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260928 02:36:49.427316 13775 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42121: connect: Connection refused (error 111)
I20260928 02:36:49.427707 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.427603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42121 (local address 127.0.0.1:46350)
0928 02:36:49.427663 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:49.427687 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 02:36:49.428460 13825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42121: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42121: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260928 02:36:49.439471 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42121
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37939,127.0.98.61:40077,127.0.98.60:42121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:36:49.477646 13825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42121 } attempt: 1
I20260928 02:36:49.478104 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.477950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42121 (local address 127.0.0.1:46362)
0928 02:36:49.478013 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:49.478063 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 02:36:49.478434 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.478348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42121 (local address 127.0.0.1:46370)
0928 02:36:49.478389 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:49.478415 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 02:36:49.478528 13825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42121: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42121: connect: Connection refused (error 111)
I20260928 02:36:49.482231 13750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40077 } attempt: 3
Sep 28 02:36:49 dist-test-slave-fz47 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 02:36:49.488132 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.482524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45724 (local address 127.0.98.61:40077)
0928 02:36:49.482574 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:49.482576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:49.482585 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:36:49.482627 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:49.482628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:49.482648 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:49.482691 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:49.482991 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.483517 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.484158 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.484359 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.484412 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:49.486022 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:49.486023 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:49.486024 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:49.486030 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:49.487209 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:49.487496 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:49.487497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:49.487498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:49.487544 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:49.487693 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:49.487694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:49.487694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:49.487801 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:49.487852 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:49.487979 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 02:36:49.488045 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 02:36:49.488137 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.482476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40077 (local address 127.0.0.1:45724)
0928 02:36:49.482516 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:49.482525 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:49.482571 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:49.482701 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:49.482702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:49.482847 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:36:49.482909 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:49.482910 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.483602 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.483603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:49.484085 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:49.484087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.484418 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.484419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:49.484486 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:49.484874 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:36:49.484879 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:36:49.485950 (+ 1071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:36:49.487287 (+ 1337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:49.487288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:49.487289 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:49.487440 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:49.487593 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:49.487594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:49.487594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:49.487647 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:49.487852 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:36:49.487853 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:36:49.487896 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 02:36:49.487954 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 02:36:49.488044 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 02:36:49.488646 13750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42121 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:36:49.488945 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.488836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42121 (local address 127.0.0.1:46380)
0928 02:36:49.488883 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:49.488918 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 02:36:49.489214 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.489135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42121 (local address 127.0.0.1:46392)
0928 02:36:49.489172 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:49.489195 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 02:36:49.489326 13750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42121: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42121: connect: Connection refused (error 111)
W20260928 02:36:49.517146 13832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:36:49.517379 13832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:36:49.517405 13832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:36:49.518487 13832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:36:49.518520 13832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:36:49.518532 13832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:36:49.518543 13832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:36:49.518560 13832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:36:49.519958 13832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37939,127.0.98.61:40077,127.0.98.60:42121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:36:49.520150 13832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:36:49.520318 13832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:36:49.520469 13750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42121 } attempt: 1
I20260928 02:36:49.520907 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.520754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42121 (local address 127.0.0.1:46402)
0928 02:36:49.520829 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:49.520873 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 02:36:49.521247 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.521150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42121 (local address 127.0.0.1:46418)
0928 02:36:49.521194 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:49.521223 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 02:36:49.521354 13750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42121: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42121: connect: Connection refused (error 111)
W20260928 02:36:49.524799 13837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:49.524889 13838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:49.524919 13840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:49.525347 13832 server_base.cc:1034] running on GCE node
I20260928 02:36:49.525543 13832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:36:49.525797 13832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:36:49.526947 13832 hybrid_clock.cc:648] HybridClock initialized: now 1790563009526928 us; error 25 us; skew 500 ppm
Sep 28 02:36:49 dist-test-slave-fz47 krb5kdc[13669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:36:49.529235 13832 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260928 02:36:49.529686 13832 webserver.cc:460] Webserver started at http://127.0.98.60:37775/ using document root <none> and password file <none>
I20260928 02:36:49.529883 13832 fs_manager.cc:359] Metadata directory not provided
I20260928 02:36:49.529929 13832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:36:49.530045 13832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:36:49.531008 13832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-2/data/instance:
uuid: "5b9773b793dc4f99aaf667322a987251"
format_stamp: "Formatted at 2026-09-28 02:36:49 on dist-test-slave-fz47"
server_key: "61c8f10762f827b010d7317750f4db1a"
server_key_iv: "ed9b97dc8f5c2729be66d2d5d1e061a5"
server_key_version: "encryptionkey@0"
I20260928 02:36:49.531342 13832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-2/wal/instance:
uuid: "5b9773b793dc4f99aaf667322a987251"
format_stamp: "Formatted at 2026-09-28 02:36:49 on dist-test-slave-fz47"
server_key: "61c8f10762f827b010d7317750f4db1a"
server_key_iv: "ed9b97dc8f5c2729be66d2d5d1e061a5"
server_key_version: "encryptionkey@0"
I20260928 02:36:49.532518 13832 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:36:49.533174 13847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:49.533334 13832 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:36:49.533413 13832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-2/wal
uuid: "5b9773b793dc4f99aaf667322a987251"
format_stamp: "Formatted at 2026-09-28 02:36:49 on dist-test-slave-fz47"
server_key: "61c8f10762f827b010d7317750f4db1a"
server_key_iv: "ed9b97dc8f5c2729be66d2d5d1e061a5"
server_key_version: "encryptionkey@0"
I20260928 02:36:49.533473 13832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:36:49.546657 13825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42121 } attempt: 2
I20260928 02:36:49.547067 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.546930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42121 (local address 127.0.0.1:46434)
0928 02:36:49.546987 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:49.547029 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 02:36:49.547369 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.547296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42121 (local address 127.0.0.1:46436)
0928 02:36:49.547330 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:49.547352 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 02:36:49.547442 13825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42121: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42121: connect: Connection refused (error 111)
I20260928 02:36:49.560943 13832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:36:49.562680 13832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:36:49.562785 13832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:36:49.566658 13832 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42121
I20260928 02:36:49.566692 13899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42121 every 8 connection(s)
I20260928 02:36:49.567003 13832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-2/data/info.pb
I20260928 02:36:49.567772 13900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:36:49.569232 13900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:36:49.573433 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 13832
I20260928 02:36:49.573535 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790562862804661-392-0/minicluster-data/master-2/wal/instance
I20260928 02:36:49.573719 392 external_mini_cluster.cc:1428] Setting key 4be2db2d48d20d9a3afd1b5d7adef130
Sep 28 02:36:49 dist-test-slave-fz47 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:36:49.578167 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.569942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55134 (local address 127.0.98.62:37939)
0928 02:36:49.570024 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:49.570027 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:49.570196 (+ 169us) server_negotiation.cc:407] Connection header received
0928 02:36:49.570655 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:49.570657 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:49.570668 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:49.570725 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:49.571325 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.571802 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.572750 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.572949 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.573003 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:49.575352 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:49.575359 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:49.575360 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:49.575366 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:49.576734 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:49.577112 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:49.577113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:49.577113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:49.577164 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:49.577431 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:49.577433 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:49.577433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:49.577600 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:49.577680 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:49.578002 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 02:36:49.578080 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:36:49.578209 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.569882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37939 (local address 127.0.0.1:55134)
0928 02:36:49.570048 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:49.570085 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:49.570558 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:49.570796 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:49.570799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:49.571038 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:36:49.571235 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:49.571239 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.571883 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.571884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:49.572668 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:49.572670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.573012 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.573013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:49.573125 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:49.573680 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:36:49.573687 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:36:49.575263 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:36:49.576801 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:49.576806 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:49.576808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:49.577047 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:49.577213 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:49.577214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:49.577214 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:49.577348 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:49.577669 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:36:49.577673 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:36:49.577831 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 02:36:49.577946 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 02:36:49.578058 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":72,"threads_started":1}
I20260928 02:36:49.578735 13900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40077 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:36:49 dist-test-slave-fz47 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 02:36:49 dist-test-slave-fz47 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 02:36:49.582518 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.574062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46438 (local address 127.0.98.60:42121)
0928 02:36:49.574189 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:49.574194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:49.574390 (+ 196us) server_negotiation.cc:407] Connection header received
0928 02:36:49.574413 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:49.574416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:49.574438 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:49.574505 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:49.575061 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.575981 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.576953 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.577406 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.577461 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:49.580015 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:49.580022 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:49.580023 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:49.580028 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:49.581472 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:49.581757 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:49.581759 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:49.581760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:49.581812 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:49.581965 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:49.581966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:49.581966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:49.582094 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:49.582290 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:49.582332 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 02:36:49.582414 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":63,"threads_started":1}
I20260928 02:36:49.584882 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.578966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40077 (local address 127.0.0.1:45736)
0928 02:36:49.579082 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:49.579105 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:49.579154 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:49.579291 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:49.579292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:49.579432 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:36:49.579483 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:49.579485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.580107 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.580108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:49.580562 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:49.580563 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.580976 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.580977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:49.581044 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:49.581422 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:36:49.581425 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:36:49.582464 (+ 1039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:36:49.584059 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:49.584060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:49.584060 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:49.584258 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:49.584413 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:49.584414 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:49.584414 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:49.584468 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:49.584652 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:36:49.584653 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:36:49.584691 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:36:49.584744 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 02:36:49.584807 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260928 02:36:49.584944 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.579036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45736 (local address 127.0.98.61:40077)
0928 02:36:49.579078 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:49.579080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:49.579139 (+ 59us) server_negotiation.cc:407] Connection header received
0928 02:36:49.579209 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:49.579210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:49.579218 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:49.579257 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:49.579548 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.580033 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.580635 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.580897 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.580967 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:49.582558 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:49.582559 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:49.582560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:49.582566 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:49.583995 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:49.584320 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:49.584321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:49.584321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:49.584365 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:49.584516 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:49.584516 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:49.584516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:49.584600 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:49.584647 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:49.584782 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 02:36:49.584854 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:36:49.585240 13900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42121 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:36:49 dist-test-slave-fz47 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 28 02:36:49 dist-test-slave-fz47 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 02:36:49.589957 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.583241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55144 (local address 127.0.98.62:37939)
0928 02:36:49.583279 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:49.583281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:49.583289 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:36:49.583340 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:49.583341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:49.583350 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:49.583400 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:49.583914 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.584594 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.585540 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.585732 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.585783 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:49.587589 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:49.587591 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:49.587591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:49.587598 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:49.588907 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:49.589209 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:49.589210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:49.589211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:49.589287 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:49.589448 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:49.589448 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:49.589449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:49.589571 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:49.589763 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:49.589803 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 02:36:49.589868 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:36:49.591118 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.585495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42121 (local address 127.0.0.1:46440)
0928 02:36:49.585519 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:49.585528 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:49.585562 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:49.585668 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:49.585668 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:49.585778 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:36:49.585801 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:49.585802 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.586495 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.586495 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:49.586944 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:49.586945 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.587259 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.587260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:49.587304 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:49.587608 (+ 304us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:36:49.587613 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:36:49.588809 (+ 1196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:36:49.590294 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:49.590295 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:49.590295 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:49.590443 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:49.590610 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:49.590611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:49.590611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:49.590658 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:49.590849 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:36:49.590850 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:36:49.590883 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:36:49.590942 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 02:36:49.591023 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12}
I20260928 02:36:49.591166 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.585523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46440 (local address 127.0.98.60:42121)
0928 02:36:49.585550 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:49.585552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:49.585559 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:36:49.585611 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:49.585611 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:49.585619 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:49.585660 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:49.585855 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.586429 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.587002 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.587206 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.587255 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:49.588879 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:49.588880 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:49.588880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:49.588886 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:49.590220 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:49.590501 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:49.590501 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:49.590502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:49.590550 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:49.590707 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:49.590708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:49.590708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:49.590800 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:49.590850 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:49.590976 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 02:36:49.591069 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":17}
I20260928 02:36:49.592274 13900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b9773b793dc4f99aaf667322a987251: Bootstrap starting.
I20260928 02:36:49.593114 13900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b9773b793dc4f99aaf667322a987251: Neither blocks nor log segments found. Creating new log.
I20260928 02:36:49.593405 13900 log.cc:824] T 00000000000000000000000000000000 P 5b9773b793dc4f99aaf667322a987251: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:36:49 dist-test-slave-fz47 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 02:36:49.594172 13900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b9773b793dc4f99aaf667322a987251: No bootstrap required, opened a new log
I20260928 02:36:49.595645 13900 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b9773b793dc4f99aaf667322a987251 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f125a383e1b64933869af0e14b436509" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "4fa5a7e6d51148c7ba73a85e581fcfb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40077 } } peers { permanent_uuid: "5b9773b793dc4f99aaf667322a987251" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42121 } }
I20260928 02:36:49.595770 13900 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b9773b793dc4f99aaf667322a987251 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:36:49.595793 13900 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b9773b793dc4f99aaf667322a987251 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b9773b793dc4f99aaf667322a987251, State: Initialized, Role: FOLLOWER
I20260928 02:36:49.595901 13900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b9773b793dc4f99aaf667322a987251 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f125a383e1b64933869af0e14b436509" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "4fa5a7e6d51148c7ba73a85e581fcfb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40077 } } peers { permanent_uuid: "5b9773b793dc4f99aaf667322a987251" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42121 } }
I20260928 02:36:49.596174 13900 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b9773b793dc4f99aaf667322a987251 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:36:49.596138 13906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b9773b793dc4f99aaf667322a987251 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f125a383e1b64933869af0e14b436509" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "4fa5a7e6d51148c7ba73a85e581fcfb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40077 } } peers { permanent_uuid: "5b9773b793dc4f99aaf667322a987251" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42121 } } }
I20260928 02:36:49.596238 13906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b9773b793dc4f99aaf667322a987251 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:36:49.596487 13750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42121 } attempt: 2
I20260928 02:36:49.596925 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.590510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45750 (local address 127.0.98.61:40077)
0928 02:36:49.590548 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:49.590550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:49.590557 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:36:49.590575 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:49.590576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:49.590585 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:49.590632 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:49.591002 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.591636 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.592215 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.592448 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.592495 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:49.594600 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:49.594601 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:49.594602 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:49.594608 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:49.595780 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:49.596135 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:49.596136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:49.596137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:49.596195 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:49.596411 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:49.596412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:49.596412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:49.596511 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:49.596662 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:49.596768 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 02:36:49.596838 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 02:36:49.597976 13917 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5b9773b793dc4f99aaf667322a987251: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:36:49.598069 13917 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:36:49.598663 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Sep 28 02:36:49 dist-test-slave-fz47 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 02:36:49.598248 13832 master_runner.cc:186] Error getting master registration for 127.0.98.62:37939: OK, instance_id { permanent_uuid: "f125a383e1b64933869af0e14b436509" instance_seqno: 1790563009257448 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:36:49.601612 13832 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:36:49.603861 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.596689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42121 (local address 127.0.0.1:46452)
0928 02:36:49.596767 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:49.596776 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:49.596810 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:49.597035 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:49.597037 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:49.597219 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:36:49.597281 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:49.597284 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.598044 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.598046 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:49.598767 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:49.598769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.599207 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.599207 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:49.599267 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:49.599609 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:36:49.599613 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:36:49.600939 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:36:49.602827 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:49.602828 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:49.602829 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:49.603026 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:49.603234 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:49.603235 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:49.603235 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:49.603299 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:49.603578 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:36:49.603580 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:36:49.603623 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 02:36:49.603690 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 02:36:49.603772 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 02:36:49.604118 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.596883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46452 (local address 127.0.98.60:42121)
0928 02:36:49.596917 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:49.596919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:49.596926 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:36:49.596943 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:49.596943 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:49.596952 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:49.597348 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:49.597375 (+ 27us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.597969 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.598857 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.599107 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.599402 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:49.601062 (+ 1660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:49.601063 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:49.601063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:49.601069 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:49.602590 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:49.603120 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:49.603121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:49.603121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:49.603172 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:49.603360 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:49.603361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:49.603361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:49.603502 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:49.603567 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:49.603969 (+ 402us) server_negotiation.cc:299] Negotiation successful
0928 02:36:49.604038 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:36:49.605993 13750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509: Bootstrap starting.
I20260928 02:36:49.606670 13750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509: Neither blocks nor log segments found. Creating new log.
I20260928 02:36:49.606947 13750 log.cc:824] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509: Log is configured to *not* fsync() on all Append() calls
I20260928 02:36:49.607767 13750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509: No bootstrap required, opened a new log
I20260928 02:36:49.609014 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.600238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45754 (local address 127.0.98.61:40077)
0928 02:36:49.600271 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:49.600272 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:49.600494 (+ 222us) server_negotiation.cc:407] Connection header received
0928 02:36:49.600577 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:49.600578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:49.600586 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:49.600611 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:49.601315 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.601939 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.602886 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.603186 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.603225 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:49.605032 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:49.605034 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:49.605034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:49.605040 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:49.606504 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:49.608291 (+ 1787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:49.608292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:49.608292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:49.608343 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:49.608509 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:49.608510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:49.608510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:49.608613 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:49.608689 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:49.608863 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 02:36:49.608931 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:36:49.609721 13750 raft_consensus.cc:348] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f125a383e1b64933869af0e14b436509" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "4fa5a7e6d51148c7ba73a85e581fcfb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40077 } } peers { permanent_uuid: "5b9773b793dc4f99aaf667322a987251" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42121 } }
I20260928 02:36:49.609843 13750 raft_consensus.cc:374] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:36:49.609874 13750 raft_consensus.cc:729] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f125a383e1b64933869af0e14b436509, State: Initialized, Role: FOLLOWER
I20260928 02:36:49.609980 13750 consensus_queue.cc:260] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f125a383e1b64933869af0e14b436509" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "4fa5a7e6d51148c7ba73a85e581fcfb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40077 } } peers { permanent_uuid: "5b9773b793dc4f99aaf667322a987251" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42121 } }
I20260928 02:36:49.610251 13750 sys_catalog.cc:564] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:36:49.610617 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.601495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55154 (local address 127.0.98.62:37939)
0928 02:36:49.601556 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:49.601558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:49.602089 (+ 531us) server_negotiation.cc:407] Connection header received
0928 02:36:49.602170 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:49.602170 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:49.602179 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:49.602204 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:49.602845 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.603702 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.604591 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.604886 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.604925 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:49.606567 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:49.606569 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:49.606569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:49.606575 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:49.608002 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:49.609818 (+ 1816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:49.609819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:49.609820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:49.609872 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:49.610071 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:49.610072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:49.610072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:49.610190 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:49.610270 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:49.610469 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 02:36:49.610536 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 02:36:49.612594 13939 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:36:49.612687 13939 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:36:49.612948 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.602712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46456 (local address 127.0.98.60:42121)
0928 02:36:49.604184 (+ 1472us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:49.604185 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:49.605451 (+ 1266us) server_negotiation.cc:407] Connection header received
0928 02:36:49.605839 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:49.605840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:49.605849 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:49.605876 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:49.607108 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.607767 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.608655 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.608908 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.608942 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:49.610482 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:49.610483 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:49.610483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:49.610489 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:49.611774 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:49.612117 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:49.612118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:49.612118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:49.612179 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:49.612384 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:49.612385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:49.612385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:49.612496 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:49.612584 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:49.612803 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 02:36:49.612866 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1461,"thread_start_us":82,"threads_started":1}
I20260928 02:36:49.613499 13682 master_runner.cc:186] Error getting master registration for 127.0.98.61:40077: OK, instance_id { permanent_uuid: "4fa5a7e6d51148c7ba73a85e581fcfb2" instance_seqno: 1790563009407396 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:36:49.616194 13682 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:36:49.620481 13928 sys_catalog.cc:455] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f125a383e1b64933869af0e14b436509" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "4fa5a7e6d51148c7ba73a85e581fcfb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40077 } } peers { permanent_uuid: "5b9773b793dc4f99aaf667322a987251" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42121 } } }
I20260928 02:36:49.620600 13928 sys_catalog.cc:458] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:36:49.638574 13825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42121 } attempt: 3
Sep 28 02:36:49 dist-test-slave-fz47 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 02:36:49.646514 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.638904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46470 (local address 127.0.98.60:42121)
0928 02:36:49.638945 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:49.638947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:49.638955 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:36:49.639003 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:49.639004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:49.639015 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:49.639065 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:49.639400 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.640210 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.641201 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.641529 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.641598 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:49.643786 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:49.643787 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:49.643787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:49.643794 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:49.645373 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:49.645755 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:49.645756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:49.645756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:49.645813 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:49.645989 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:49.645989 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:49.645990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:49.646124 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:49.646185 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:49.646337 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 02:36:49.646420 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:36:49.646510 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:49.638851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42121 (local address 127.0.0.1:46470)
0928 02:36:49.638904 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:49.638914 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:49.638948 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:49.639082 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:49.639083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:49.639263 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:36:49.639313 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:49.639315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.640297 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.640299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:49.641109 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:49.641121 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:49.641599 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:49.641601 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:49.641659 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:49.642161 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:36:49.642166 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:36:49.643704 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:36:49.645464 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:49.645465 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:49.645466 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:49.645682 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:49.645871 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:49.645872 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:49.645872 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:49.645935 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:49.646180 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:36:49.646181 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:36:49.646228 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 02:36:49.646293 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 02:36:49.646398 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 02:36:49.648437 13825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fa5a7e6d51148c7ba73a85e581fcfb2: Bootstrap starting.
I20260928 02:36:49.649179 13825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fa5a7e6d51148c7ba73a85e581fcfb2: Neither blocks nor log segments found. Creating new log.
I20260928 02:36:49.649509 13825 log.cc:824] T 00000000000000000000000000000000 P 4fa5a7e6d51148c7ba73a85e581fcfb2: Log is configured to *not* fsync() on all Append() calls
I20260928 02:36:49.650830 13825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fa5a7e6d51148c7ba73a85e581fcfb2: No bootstrap required, opened a new log
I20260928 02:36:49.652820 13825 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fa5a7e6d51148c7ba73a85e581fcfb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f125a383e1b64933869af0e14b436509" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "4fa5a7e6d51148c7ba73a85e581fcfb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40077 } } peers { permanent_uuid: "5b9773b793dc4f99aaf667322a987251" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42121 } }
I20260928 02:36:49.652961 13825 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fa5a7e6d51148c7ba73a85e581fcfb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:36:49.652993 13825 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fa5a7e6d51148c7ba73a85e581fcfb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fa5a7e6d51148c7ba73a85e581fcfb2, State: Initialized, Role: FOLLOWER
I20260928 02:36:49.653174 13825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fa5a7e6d51148c7ba73a85e581fcfb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f125a383e1b64933869af0e14b436509" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "4fa5a7e6d51148c7ba73a85e581fcfb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40077 } } peers { permanent_uuid: "5b9773b793dc4f99aaf667322a987251" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42121 } }
I20260928 02:36:49.653564 13825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fa5a7e6d51148c7ba73a85e581fcfb2 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 02:36:49.655332 13778 tablet.cc:2369] T 00000000000000000000000000000000 P 4fa5a7e6d51148c7ba73a85e581fcfb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:36:49.655377 13953 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4fa5a7e6d51148c7ba73a85e581fcfb2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:36:49.655421 13953 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:36:49.653517 13942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fa5a7e6d51148c7ba73a85e581fcfb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f125a383e1b64933869af0e14b436509" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "4fa5a7e6d51148c7ba73a85e581fcfb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40077 } } peers { permanent_uuid: "5b9773b793dc4f99aaf667322a987251" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42121 } } }
I20260928 02:36:49.657703 13942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fa5a7e6d51148c7ba73a85e581fcfb2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:36:49.657902 13757 master_runner.cc:263] No leader master found from master 4fa5a7e6d51148c7ba73a85e581fcfb2
I20260928 02:36:49.657936 13757 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:36:49.724853 13928 raft_consensus.cc:482] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:36:49.724978 13928 raft_consensus.cc:504] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f125a383e1b64933869af0e14b436509" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "4fa5a7e6d51148c7ba73a85e581fcfb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40077 } } peers { permanent_uuid: "5b9773b793dc4f99aaf667322a987251" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42121 } }
I20260928 02:36:49.725216 13928 leader_election.cc:288] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fa5a7e6d51148c7ba73a85e581fcfb2 (127.0.98.61:40077), 5b9773b793dc4f99aaf667322a987251 (127.0.98.60:42121)
I20260928 02:36:49.725838 13874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f125a383e1b64933869af0e14b436509" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b9773b793dc4f99aaf667322a987251" is_pre_election: true
I20260928 02:36:49.725994 13874 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5b9773b793dc4f99aaf667322a987251 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f125a383e1b64933869af0e14b436509 in term 0.
I20260928 02:36:49.726302 13700 leader_election.cc:302] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b9773b793dc4f99aaf667322a987251, f125a383e1b64933869af0e14b436509; no voters:
I20260928 02:36:49.726487 13928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:36:49.726537 13928 raft_consensus.cc:482] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:36:49.726558 13928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:36:49.727456 13928 raft_consensus.cc:504] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f125a383e1b64933869af0e14b436509" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "4fa5a7e6d51148c7ba73a85e581fcfb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40077 } } peers { permanent_uuid: "5b9773b793dc4f99aaf667322a987251" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42121 } }
I20260928 02:36:49.727573 13928 leader_election.cc:288] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509 [CANDIDATE]: Term 1 election: Requested vote from peers 4fa5a7e6d51148c7ba73a85e581fcfb2 (127.0.98.61:40077), 5b9773b793dc4f99aaf667322a987251 (127.0.98.60:42121)
I20260928 02:36:49.727895 13874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f125a383e1b64933869af0e14b436509" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b9773b793dc4f99aaf667322a987251"
I20260928 02:36:49.727967 13874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b9773b793dc4f99aaf667322a987251 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:36:49.728765 13874 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5b9773b793dc4f99aaf667322a987251 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f125a383e1b64933869af0e14b436509 in term 1.
I20260928 02:36:49.728966 13700 leader_election.cc:302] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b9773b793dc4f99aaf667322a987251, f125a383e1b64933869af0e14b436509; no voters:
I20260928 02:36:49.730070 13799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f125a383e1b64933869af0e14b436509" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fa5a7e6d51148c7ba73a85e581fcfb2" is_pre_election: true
I20260928 02:36:49.730682 13799 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4fa5a7e6d51148c7ba73a85e581fcfb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f125a383e1b64933869af0e14b436509 in term 0.
I20260928 02:36:49.730072 13798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f125a383e1b64933869af0e14b436509" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fa5a7e6d51148c7ba73a85e581fcfb2"
I20260928 02:36:49.730831 13798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fa5a7e6d51148c7ba73a85e581fcfb2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:36:49.731567 13798 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4fa5a7e6d51148c7ba73a85e581fcfb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f125a383e1b64933869af0e14b436509 in term 1.
I20260928 02:36:49.732064 13928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:36:49.732191 13928 raft_consensus.cc:686] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509 [term 1 LEADER]: Becoming Leader. State: Replica: f125a383e1b64933869af0e14b436509, State: Running, Role: LEADER
I20260928 02:36:49.732272 13928 consensus_queue.cc:237] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f125a383e1b64933869af0e14b436509" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "4fa5a7e6d51148c7ba73a85e581fcfb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40077 } } peers { permanent_uuid: "5b9773b793dc4f99aaf667322a987251" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42121 } }
I20260928 02:36:49.732669 13928 sys_catalog.cc:455] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f125a383e1b64933869af0e14b436509. Latest consensus state: current_term: 1 leader_uuid: "f125a383e1b64933869af0e14b436509" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f125a383e1b64933869af0e14b436509" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "4fa5a7e6d51148c7ba73a85e581fcfb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40077 } } peers { permanent_uuid: "5b9773b793dc4f99aaf667322a987251" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42121 } } }
I20260928 02:36:49.732735 13928 sys_catalog.cc:458] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509 [sys.catalog]: This master's current role is: LEADER
I20260928 02:36:49.733067 13957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:36:49.733328 13957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:36:49.735653 13874 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5b9773b793dc4f99aaf667322a987251 [term 1 FOLLOWER]: Refusing update from remote peer f125a383e1b64933869af0e14b436509: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:36:49.736073 13955 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b9773b793dc4f99aaf667322a987251" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:36:49.737409 13906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b9773b793dc4f99aaf667322a987251 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f125a383e1b64933869af0e14b436509. Latest consensus state: current_term: 1 leader_uuid: "f125a383e1b64933869af0e14b436509" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f125a383e1b64933869af0e14b436509" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "4fa5a7e6d51148c7ba73a85e581fcfb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40077 } } peers { permanent_uuid: "5b9773b793dc4f99aaf667322a987251" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42121 } } }
I20260928 02:36:49.737493 13906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b9773b793dc4f99aaf667322a987251 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:36:49.737871 13798 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4fa5a7e6d51148c7ba73a85e581fcfb2 [term 1 FOLLOWER]: Refusing update from remote peer f125a383e1b64933869af0e14b436509: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:36:49.738116 13955 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fa5a7e6d51148c7ba73a85e581fcfb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:36:49.739090 13942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fa5a7e6d51148c7ba73a85e581fcfb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f125a383e1b64933869af0e14b436509. Latest consensus state: current_term: 1 leader_uuid: "f125a383e1b64933869af0e14b436509" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f125a383e1b64933869af0e14b436509" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "4fa5a7e6d51148c7ba73a85e581fcfb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40077 } } peers { permanent_uuid: "5b9773b793dc4f99aaf667322a987251" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42121 } } }
I20260928 02:36:49.739164 13942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fa5a7e6d51148c7ba73a85e581fcfb2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:36:49.739996 13955 sys_catalog.cc:455] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f125a383e1b64933869af0e14b436509" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f125a383e1b64933869af0e14b436509" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "4fa5a7e6d51148c7ba73a85e581fcfb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40077 } } peers { permanent_uuid: "5b9773b793dc4f99aaf667322a987251" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42121 } } }
I20260928 02:36:49.740067 13955 sys_catalog.cc:458] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509 [sys.catalog]: This master's current role is: LEADER
I20260928 02:36:49.740429 13942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fa5a7e6d51148c7ba73a85e581fcfb2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f125a383e1b64933869af0e14b436509" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f125a383e1b64933869af0e14b436509" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "4fa5a7e6d51148c7ba73a85e581fcfb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40077 } } peers { permanent_uuid: "5b9773b793dc4f99aaf667322a987251" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42121 } } }
I20260928 02:36:49.740496 13942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fa5a7e6d51148c7ba73a85e581fcfb2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:36:49.740701 13958 mvcc.cc:205] Tried to move back new op lower bound from 7334146087874887680 to 7334146087864152064. Current Snapshot: MvccSnapshot[applied={T|T < 7334146087874887680}]
I20260928 02:36:49.741153 13961 mvcc.cc:205] Tried to move back new op lower bound from 7334146087874887680 to 7334146087864152064. Current Snapshot: MvccSnapshot[applied={T|T < 7334146087874887680}]
I20260928 02:36:49.742061 13928 sys_catalog.cc:455] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f125a383e1b64933869af0e14b436509" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f125a383e1b64933869af0e14b436509" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "4fa5a7e6d51148c7ba73a85e581fcfb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40077 } } peers { permanent_uuid: "5b9773b793dc4f99aaf667322a987251" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42121 } } }
I20260928 02:36:49.742132 13928 sys_catalog.cc:458] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509 [sys.catalog]: This master's current role is: LEADER
I20260928 02:36:49.742539 13906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b9773b793dc4f99aaf667322a987251 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f125a383e1b64933869af0e14b436509" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f125a383e1b64933869af0e14b436509" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37939 } } peers { permanent_uuid: "4fa5a7e6d51148c7ba73a85e581fcfb2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40077 } } peers { permanent_uuid: "5b9773b793dc4f99aaf667322a987251" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42121 } } }
I20260928 02:36:49.742612 13906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b9773b793dc4f99aaf667322a987251 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:36:49.742866 13959 mvcc.cc:205] Tried to move back new op lower bound from 7334146087874887680 to 7334146087864152064. Current Snapshot: MvccSnapshot[applied={T|T < 7334146087874887680}]
I20260928 02:36:49.743408 13957 catalog_manager.cc:1347] Generated new cluster ID: 3a0ef95bc9d940bb863ffb40c6039709
I20260928 02:36:49.743446 13957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:36:49.767544 13957 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:36:49.768200 13957 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:36:49.773187 13957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f125a383e1b64933869af0e14b436509: Generated new TSK 0
I20260928 02:36:49.773425 13957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:36:50.598534 13917 catalog_manager.cc:1259] Loaded cluster ID: 3a0ef95bc9d940bb863ffb40c6039709
I20260928 02:36:50.598609 13917 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5b9773b793dc4f99aaf667322a987251: loading cluster ID for follower catalog manager: success
I20260928 02:36:50.599421 13917 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5b9773b793dc4f99aaf667322a987251: acquiring CA information for follower catalog manager: success
I20260928 02:36:50.599692 13917 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5b9773b793dc4f99aaf667322a987251: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:36:50.655802 13953 catalog_manager.cc:1259] Loaded cluster ID: 3a0ef95bc9d940bb863ffb40c6039709
I20260928 02:36:50.655870 13953 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4fa5a7e6d51148c7ba73a85e581fcfb2: loading cluster ID for follower catalog manager: success
I20260928 02:36:50.656672 13953 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4fa5a7e6d51148c7ba73a85e581fcfb2: acquiring CA information for follower catalog manager: success
I20260928 02:36:50.656962 13953 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4fa5a7e6d51148c7ba73a85e581fcfb2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:36:50.824189 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 13682
I20260928 02:36:50.828825 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 13757
I20260928 02:36:50.832897 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 13832
2026-09-28T02:36:50Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2742 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:36:50 dist-test-slave-fz47 krb5kdc[13972](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:36:50 dist-test-slave-fz47 krb5kdc[13972](info): set up 2 sockets
Sep 28 02:36:50 dist-test-slave-fz47 krb5kdc[13972](info): commencing operation
krb5kdc: starting...
W20260928 02:36:51.788971 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:36:51 dist-test-slave-fz47 krb5kdc[13972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:36:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:36:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 02:36:51.896034 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44995
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:36:51.978547 13985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:36:51.978740 13985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:36:51.978770 13985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:36:51.979866 13985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:36:51.979902 13985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:36:51.979921 13985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:36:51.979945 13985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:36:51.979961 13985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:36:51.981410 13985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:36:51.981638 13985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:36:51.981827 13985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:36:51.986312 13991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:51.986330 13985 server_base.cc:1034] running on GCE node
W20260928 02:36:51.986443 13993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:51.986416 13990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:51.986733 13985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:36:51.986963 13985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:36:51.988106 13985 hybrid_clock.cc:648] HybridClock initialized: now 1790563011988090 us; error 26 us; skew 500 ppm
Sep 28 02:36:51 dist-test-slave-fz47 krb5kdc[13972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:36:51.990345 13985 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:36:51.990746 13985 webserver.cc:460] Webserver started at http://127.0.98.62:37813/ using document root <none> and password file <none>
I20260928 02:36:51.990932 13985 fs_manager.cc:359] Metadata directory not provided
I20260928 02:36:51.990981 13985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:36:51.991099 13985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:36:51.992095 13985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "be8c916c9d1a47af9b55d492923551eb"
format_stamp: "Formatted at 2026-09-28 02:36:51 on dist-test-slave-fz47"
server_key: "e71054c8690cdf9db80f059cc5c2b288"
server_key_iv: "75eb7dd9cae83c322ce3df5c9e4ff3a7"
server_key_version: "encryptionkey@0"
I20260928 02:36:51.992406 13985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "be8c916c9d1a47af9b55d492923551eb"
format_stamp: "Formatted at 2026-09-28 02:36:51 on dist-test-slave-fz47"
server_key: "e71054c8690cdf9db80f059cc5c2b288"
server_key_iv: "75eb7dd9cae83c322ce3df5c9e4ff3a7"
server_key_version: "encryptionkey@0"
I20260928 02:36:51.993615 13985 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:36:51.994230 14000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:51.994372 13985 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:36:51.994455 13985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "be8c916c9d1a47af9b55d492923551eb"
format_stamp: "Formatted at 2026-09-28 02:36:51 on dist-test-slave-fz47"
server_key: "e71054c8690cdf9db80f059cc5c2b288"
server_key_iv: "75eb7dd9cae83c322ce3df5c9e4ff3a7"
server_key_version: "encryptionkey@0"
I20260928 02:36:51.994510 13985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:36:52.009771 13985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:36:52.011397 13985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:36:52.011523 13985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:36:52.015100 13985 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44995
I20260928 02:36:52.015138 14052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44995 every 8 connection(s)
I20260928 02:36:52.015439 13985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:36:52.016192 14053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:36:52.018316 14053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be8c916c9d1a47af9b55d492923551eb: Bootstrap starting.
I20260928 02:36:52.018806 14053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be8c916c9d1a47af9b55d492923551eb: Neither blocks nor log segments found. Creating new log.
I20260928 02:36:52.019022 14053 log.cc:824] T 00000000000000000000000000000000 P be8c916c9d1a47af9b55d492923551eb: Log is configured to *not* fsync() on all Append() calls
I20260928 02:36:52.019579 14053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be8c916c9d1a47af9b55d492923551eb: No bootstrap required, opened a new log
I20260928 02:36:52.019994 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 13985
I20260928 02:36:52.020092 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:36:52.020283 392 external_mini_cluster.cc:1428] Setting key cd3a7ee24326f5b792252fb6efe898a2
I20260928 02:36:52.021507 14053 raft_consensus.cc:348] T 00000000000000000000000000000000 P be8c916c9d1a47af9b55d492923551eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8c916c9d1a47af9b55d492923551eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44995 } }
I20260928 02:36:52.021647 14053 raft_consensus.cc:374] T 00000000000000000000000000000000 P be8c916c9d1a47af9b55d492923551eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:36:52.021679 14053 raft_consensus.cc:729] T 00000000000000000000000000000000 P be8c916c9d1a47af9b55d492923551eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be8c916c9d1a47af9b55d492923551eb, State: Initialized, Role: FOLLOWER
I20260928 02:36:52.021847 14053 consensus_queue.cc:260] T 00000000000000000000000000000000 P be8c916c9d1a47af9b55d492923551eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8c916c9d1a47af9b55d492923551eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44995 } }
I20260928 02:36:52.021927 14053 raft_consensus.cc:388] T 00000000000000000000000000000000 P be8c916c9d1a47af9b55d492923551eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:36:52.021962 14053 raft_consensus.cc:482] T 00000000000000000000000000000000 P be8c916c9d1a47af9b55d492923551eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:36:52.022007 14053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be8c916c9d1a47af9b55d492923551eb [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:36:52.022867 14053 raft_consensus.cc:504] T 00000000000000000000000000000000 P be8c916c9d1a47af9b55d492923551eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8c916c9d1a47af9b55d492923551eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44995 } }
I20260928 02:36:52.022971 14053 leader_election.cc:302] T 00000000000000000000000000000000 P be8c916c9d1a47af9b55d492923551eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be8c916c9d1a47af9b55d492923551eb; no voters:
I20260928 02:36:52.023113 14053 leader_election.cc:288] T 00000000000000000000000000000000 P be8c916c9d1a47af9b55d492923551eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:36:52.023161 14058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be8c916c9d1a47af9b55d492923551eb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:36:52.023334 14058 raft_consensus.cc:686] T 00000000000000000000000000000000 P be8c916c9d1a47af9b55d492923551eb [term 1 LEADER]: Becoming Leader. State: Replica: be8c916c9d1a47af9b55d492923551eb, State: Running, Role: LEADER
I20260928 02:36:52.023347 14053 sys_catalog.cc:564] T 00000000000000000000000000000000 P be8c916c9d1a47af9b55d492923551eb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:36:52.023461 14058 consensus_queue.cc:237] T 00000000000000000000000000000000 P be8c916c9d1a47af9b55d492923551eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8c916c9d1a47af9b55d492923551eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44995 } }
I20260928 02:36:52.023895 14060 sys_catalog.cc:455] T 00000000000000000000000000000000 P be8c916c9d1a47af9b55d492923551eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader be8c916c9d1a47af9b55d492923551eb. Latest consensus state: current_term: 1 leader_uuid: "be8c916c9d1a47af9b55d492923551eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8c916c9d1a47af9b55d492923551eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44995 } } }
I20260928 02:36:52.023916 14059 sys_catalog.cc:455] T 00000000000000000000000000000000 P be8c916c9d1a47af9b55d492923551eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be8c916c9d1a47af9b55d492923551eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8c916c9d1a47af9b55d492923551eb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44995 } } }
I20260928 02:36:52.023977 14060 sys_catalog.cc:458] T 00000000000000000000000000000000 P be8c916c9d1a47af9b55d492923551eb [sys.catalog]: This master's current role is: LEADER
I20260928 02:36:52.023994 14059 sys_catalog.cc:458] T 00000000000000000000000000000000 P be8c916c9d1a47af9b55d492923551eb [sys.catalog]: This master's current role is: LEADER
I20260928 02:36:52.024704 14067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:36:52.025200 14067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:36:52.026726 14067 catalog_manager.cc:1347] Generated new cluster ID: 26367460076740538f66bf499a15b7e4
I20260928 02:36:52.026778 14067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:36:52 dist-test-slave-fz47 krb5kdc[13972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:36:52.032104 14057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:52.020740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43952 (local address 127.0.98.62:44995)
0928 02:36:52.020891 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:52.020899 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:52.020922 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:36:52.021496 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:52.021501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:52.021546 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:52.021653 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:52.022378 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:52.023435 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:52.024413 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:52.025005 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:52.025106 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:52.028157 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:52.028160 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:52.028161 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:52.028172 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:52.030056 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:52.030463 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:52.030466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:52.030467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:52.030942 (+ 475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:52.031161 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:52.031162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:52.031162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:52.031456 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:52.031614 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:52.031766 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 02:36:52.031896 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260928 02:36:52.032699 14067 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:36:52.032897 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:36:52.033388 14067 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:36:52.038179 14067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be8c916c9d1a47af9b55d492923551eb: Generated new TSK 0
I20260928 02:36:52.038321 14067 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:36:52.086328 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 13985
2026-09-28T02:36:52Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1252 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:36:52 dist-test-slave-fz47 krb5kdc[14080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:36:52 dist-test-slave-fz47 krb5kdc[14080](info): set up 2 sockets
Sep 28 02:36:52 dist-test-slave-fz47 krb5kdc[14080](info): commencing operation
krb5kdc: starting...
W20260928 02:36:53.041142 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:36:53 dist-test-slave-fz47 krb5kdc[14080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:36:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:36:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 02:36:53.152091 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33383
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:36:53.230299 14093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:36:53.230504 14093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:36:53.230532 14093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:36:53.231737 14093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:36:53.231784 14093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:36:53.231802 14093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:36:53.231818 14093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:36:53.231832 14093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:36:53.233436 14093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:36:53.233676 14093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:36:53.233893 14093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:36:53.238408 14098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:53.238453 14101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:53.238431 14099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:53.238683 14093 server_base.cc:1034] running on GCE node
I20260928 02:36:53.238899 14093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:36:53.239171 14093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:36:53.240314 14093 hybrid_clock.cc:648] HybridClock initialized: now 1790563013240291 us; error 35 us; skew 500 ppm
Sep 28 02:36:53 dist-test-slave-fz47 krb5kdc[14080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:36:53.242882 14093 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:36:53.243330 14093 webserver.cc:460] Webserver started at http://127.0.98.62:43781/ using document root <none> and password file <none>
I20260928 02:36:53.243521 14093 fs_manager.cc:359] Metadata directory not provided
I20260928 02:36:53.243563 14093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:36:53.243687 14093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:36:53.244755 14093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "34384d463a834c0ea4abe376446aab13"
format_stamp: "Formatted at 2026-09-28 02:36:53 on dist-test-slave-fz47"
server_key: "a502da60ccceaebb557b50273b4101b8"
server_key_iv: "b9cc61d58beb3be4c243215166552638"
server_key_version: "encryptionkey@0"
I20260928 02:36:53.245080 14093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "34384d463a834c0ea4abe376446aab13"
format_stamp: "Formatted at 2026-09-28 02:36:53 on dist-test-slave-fz47"
server_key: "a502da60ccceaebb557b50273b4101b8"
server_key_iv: "b9cc61d58beb3be4c243215166552638"
server_key_version: "encryptionkey@0"
I20260928 02:36:53.246335 14093 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:36:53.247015 14108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:53.247150 14093 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:36:53.247231 14093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "34384d463a834c0ea4abe376446aab13"
format_stamp: "Formatted at 2026-09-28 02:36:53 on dist-test-slave-fz47"
server_key: "a502da60ccceaebb557b50273b4101b8"
server_key_iv: "b9cc61d58beb3be4c243215166552638"
server_key_version: "encryptionkey@0"
I20260928 02:36:53.247304 14093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:36:53.259402 14093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:36:53.261044 14093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:36:53.261168 14093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:36:53.264858 14093 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33383
I20260928 02:36:53.264887 14160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33383 every 8 connection(s)
I20260928 02:36:53.265196 14093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:36:53.266117 14161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:36:53.266134 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 14093
I20260928 02:36:53.266218 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:36:53.266397 392 external_mini_cluster.cc:1428] Setting key 8f28f04ae6e484917f517a0d116b2b92
I20260928 02:36:53.269042 14161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13: Bootstrap starting.
I20260928 02:36:53.269690 14161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13: Neither blocks nor log segments found. Creating new log.
I20260928 02:36:53.269969 14161 log.cc:824] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13: Log is configured to *not* fsync() on all Append() calls
I20260928 02:36:53.270578 14161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13: No bootstrap required, opened a new log
I20260928 02:36:53.272501 14161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34384d463a834c0ea4abe376446aab13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33383 } }
I20260928 02:36:53.272665 14161 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Sep 28 02:36:53 dist-test-slave-fz47 krb5kdc[14080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:36:53.272696 14161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34384d463a834c0ea4abe376446aab13, State: Initialized, Role: FOLLOWER
I20260928 02:36:53.272825 14161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34384d463a834c0ea4abe376446aab13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33383 } }
I20260928 02:36:53.272900 14161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:36:53.272928 14161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:36:53.272975 14161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:36:53.273561 14161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34384d463a834c0ea4abe376446aab13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33383 } }
I20260928 02:36:53.273671 14161 leader_election.cc:302] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34384d463a834c0ea4abe376446aab13; no voters:
I20260928 02:36:53.273826 14161 leader_election.cc:288] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:36:53.273862 14166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:36:53.273986 14166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [term 1 LEADER]: Becoming Leader. State: Replica: 34384d463a834c0ea4abe376446aab13, State: Running, Role: LEADER
I20260928 02:36:53.274109 14161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:36:53.274138 14166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34384d463a834c0ea4abe376446aab13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33383 } }
I20260928 02:36:53.274595 14168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34384d463a834c0ea4abe376446aab13. Latest consensus state: current_term: 1 leader_uuid: "34384d463a834c0ea4abe376446aab13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34384d463a834c0ea4abe376446aab13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33383 } } }
I20260928 02:36:53.274681 14168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [sys.catalog]: This master's current role is: LEADER
I20260928 02:36:53.274864 14175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:36:53.275398 14175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:36:53.275689 14167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34384d463a834c0ea4abe376446aab13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34384d463a834c0ea4abe376446aab13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33383 } } }
I20260928 02:36:53.275795 14167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [sys.catalog]: This master's current role is: LEADER
I20260928 02:36:53.276412 14164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:53.266872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39406 (local address 127.0.98.62:33383)
0928 02:36:53.267035 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:53.267044 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:53.267071 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:36:53.267528 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:53.267535 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:53.267580 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:53.267701 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:53.268411 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:53.269374 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:53.270323 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:53.270763 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:53.270856 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:53.273123 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:53.273128 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:53.273128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:53.273139 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:53.274922 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:53.275282 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:53.275285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:53.275286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:53.275373 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:53.275589 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:53.275590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:53.275591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:53.275881 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:53.276018 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:53.276118 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 02:36:53.276217 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
I20260928 02:36:53.278249 14175 catalog_manager.cc:1347] Generated new cluster ID: 7ab64556213d428eaaf93afd40278b18
I20260928 02:36:53.278288 14175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:36:53.303789 14175 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:36:53.304347 14175 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:36:53.308090 14175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13: Generated new TSK 0
I20260928 02:36:53.308255 14175 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 02:36:53.345067 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33383
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:36:53.425789 14187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:36:53.425979 14187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:36:53.426002 14187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:36:53.427062 14187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:36:53.427094 14187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:36:53.427119 14187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:36:53.428453 14187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:36:53.428629 14187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:36:53.428810 14187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:36:53.433351 14195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:53.433362 14193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:53.433404 14192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:53.433586 14187 server_base.cc:1034] running on GCE node
I20260928 02:36:53.433772 14187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:36:53.433970 14187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:36:53.435108 14187 hybrid_clock.cc:648] HybridClock initialized: now 1790563013435097 us; error 28 us; skew 500 ppm
Sep 28 02:36:53 dist-test-slave-fz47 krb5kdc[14080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:36:53.437408 14187 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 02:36:53.437834 14187 webserver.cc:460] Webserver started at http://127.0.98.1:44669/ using document root <none> and password file <none>
I20260928 02:36:53.438020 14187 fs_manager.cc:359] Metadata directory not provided
I20260928 02:36:53.438064 14187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:36:53.438184 14187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:36:53.439147 14187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/data/instance:
uuid: "589eafa5db924d669a9261c74dce4cb0"
format_stamp: "Formatted at 2026-09-28 02:36:53 on dist-test-slave-fz47"
server_key: "dd33b1baca925e99708089302a005fae"
server_key_iv: "39c70270a11f9d4ef3fdac9f24aa809c"
server_key_version: "encryptionkey@0"
I20260928 02:36:53.439463 14187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/wal/instance:
uuid: "589eafa5db924d669a9261c74dce4cb0"
format_stamp: "Formatted at 2026-09-28 02:36:53 on dist-test-slave-fz47"
server_key: "dd33b1baca925e99708089302a005fae"
server_key_iv: "39c70270a11f9d4ef3fdac9f24aa809c"
server_key_version: "encryptionkey@0"
I20260928 02:36:53.440604 14187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:36:53.441242 14202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:53.441406 14187 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:53.441483 14187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "589eafa5db924d669a9261c74dce4cb0"
format_stamp: "Formatted at 2026-09-28 02:36:53 on dist-test-slave-fz47"
server_key: "dd33b1baca925e99708089302a005fae"
server_key_iv: "39c70270a11f9d4ef3fdac9f24aa809c"
server_key_version: "encryptionkey@0"
I20260928 02:36:53.441545 14187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:36:53.455515 14187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:36:53.457556 14187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:36:53.457684 14187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:36:53.457878 14187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:36:53.458151 14187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:36:53.458182 14187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:53.458209 14187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:36:53.458226 14187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:53.463482 14187 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37061
I20260928 02:36:53.463538 14315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37061 every 8 connection(s)
I20260928 02:36:53.463807 14187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 02:36:53 dist-test-slave-fz47 krb5kdc[14080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:36:53.469359 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 14187
I20260928 02:36:53.469457 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/wal/instance
I20260928 02:36:53.469652 392 external_mini_cluster.cc:1428] Setting key f7199b90e0b874b35aaaa31a002a7584
I20260928 02:36:53.472287 14164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:53.464398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55941 (local address 127.0.98.62:33383)
0928 02:36:53.464437 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:53.464439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:53.464638 (+ 199us) server_negotiation.cc:407] Connection header received
0928 02:36:53.465163 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:53.465165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:53.465177 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:53.465226 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:53.466131 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:53.466632 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:53.467583 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:53.467787 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:53.467840 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:53.469849 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:53.469851 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:53.469851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:53.469858 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:53.471112 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:53.471458 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:53.471459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:53.471460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:53.471511 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:53.471713 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:53.471714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:53.471714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:53.471823 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:53.471876 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:53.472120 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 02:36:53.472200 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:36:53.472443 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:53.464376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.1:55941)
0928 02:36:53.464529 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:53.464562 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:53.465068 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:53.465325 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:53.465329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:53.465848 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:36:53.466040 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:53.466044 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:53.466711 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:53.466712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:53.467505 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:53.467507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:53.467851 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:53.467852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:53.467974 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:53.468394 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:36:53.468402 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:36:53.469771 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:36:53.471183 (+ 1412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:53.471189 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:53.471191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:53.471397 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:53.471564 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:53.471564 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:53.471565 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:53.471660 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:53.471875 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:36:53.471880 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:36:53.472009 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 02:36:53.472093 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 02:36:53.472232 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
I20260928 02:36:53.473035 14316 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33383
I20260928 02:36:53.473116 14316 heartbeater.cc:464] Registering TS with master...
I20260928 02:36:53.473304 14316 heartbeater.cc:511] Master 127.0.98.62:33383 requested a full tablet report, sending...
I20260928 02:36:53.473811 14125 ts_manager.cc:194] Registered new tserver with Master: 589eafa5db924d669a9261c74dce4cb0 (127.0.98.1:37061)
I20260928 02:36:53.474507 14125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55941
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 02:36:53.491530 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33383
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:36:53.569067 14321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:36:53.569248 14321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:36:53.569286 14321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:36:53.570374 14321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:36:53.570413 14321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:36:53.570438 14321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:36:53.571833 14321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:36:53.572038 14321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:36:53.572235 14321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:36:53.576952 14326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:53.577062 14327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:53.577071 14329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:53.577319 14321 server_base.cc:1034] running on GCE node
I20260928 02:36:53.577476 14321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:36:53.577656 14321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:36:53.578781 14321 hybrid_clock.cc:648] HybridClock initialized: now 1790563013578770 us; error 23 us; skew 500 ppm
Sep 28 02:36:53 dist-test-slave-fz47 krb5kdc[14080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:36:53.580993 14321 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 02:36:53.581416 14321 webserver.cc:460] Webserver started at http://127.0.98.2:36877/ using document root <none> and password file <none>
I20260928 02:36:53.581606 14321 fs_manager.cc:359] Metadata directory not provided
I20260928 02:36:53.581643 14321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:36:53.581737 14321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:36:53.582686 14321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/data/instance:
uuid: "ffe57bc5a65c4841aac9b43bf3205766"
format_stamp: "Formatted at 2026-09-28 02:36:53 on dist-test-slave-fz47"
server_key: "237f17fa9fef4f79568b239884002003"
server_key_iv: "b3de135f1dc25a5092746ea5ef887817"
server_key_version: "encryptionkey@0"
I20260928 02:36:53.582967 14321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ffe57bc5a65c4841aac9b43bf3205766"
format_stamp: "Formatted at 2026-09-28 02:36:53 on dist-test-slave-fz47"
server_key: "237f17fa9fef4f79568b239884002003"
server_key_iv: "b3de135f1dc25a5092746ea5ef887817"
server_key_version: "encryptionkey@0"
I20260928 02:36:53.584060 14321 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:36:53.584625 14336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:53.584765 14321 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:36:53.584841 14321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "ffe57bc5a65c4841aac9b43bf3205766"
format_stamp: "Formatted at 2026-09-28 02:36:53 on dist-test-slave-fz47"
server_key: "237f17fa9fef4f79568b239884002003"
server_key_iv: "b3de135f1dc25a5092746ea5ef887817"
server_key_version: "encryptionkey@0"
I20260928 02:36:53.584892 14321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:36:53.591913 14321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:36:53.593470 14321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:36:53.593569 14321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:36:53.593721 14321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:36:53.593945 14321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:36:53.593973 14321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:53.593992 14321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:36:53.594008 14321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:53.599113 14321 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36069
I20260928 02:36:53.599172 14449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36069 every 8 connection(s)
I20260928 02:36:53.599431 14321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 02:36:53 dist-test-slave-fz47 krb5kdc[14080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:36:53.605443 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 14321
I20260928 02:36:53.605554 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/wal/instance
I20260928 02:36:53.605712 392 external_mini_cluster.cc:1428] Setting key 09553dd0b5c565537ca109b2ae2a0a29
I20260928 02:36:53.608449 14164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:53.600022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44159 (local address 127.0.98.62:33383)
0928 02:36:53.600104 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:53.600106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:53.600260 (+ 154us) server_negotiation.cc:407] Connection header received
0928 02:36:53.600732 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:53.600734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:53.600745 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:53.600801 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:53.601723 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:53.602283 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:53.603243 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:53.603448 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:53.603509 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:53.605495 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:53.605497 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:53.605497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:53.605504 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:53.607008 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:53.607504 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:53.607505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:53.607505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:53.607567 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:53.607835 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:53.607836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:53.607836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:53.607953 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:53.608002 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:53.608310 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 02:36:53.608384 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:36:53.608670 14452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:53.600000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.2:44159)
0928 02:36:53.600155 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:53.600188 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:53.600644 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:53.600866 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:53.600870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:53.601444 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:36:53.601635 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:53.601640 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:53.602362 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:53.602363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:53.603166 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:53.603168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:53.603520 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:53.603520 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:53.603633 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:53.604047 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:36:53.604055 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:36:53.605406 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:36:53.607105 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:53.607111 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:53.607113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:53.607422 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:53.607625 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:53.607625 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:53.607625 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:53.607765 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:53.608005 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:36:53.608009 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:36:53.608165 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 02:36:53.608272 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 02:36:53.608430 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260928 02:36:53.609192 14450 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33383
I20260928 02:36:53.609351 14450 heartbeater.cc:464] Registering TS with master...
I20260928 02:36:53.609526 14450 heartbeater.cc:511] Master 127.0.98.62:33383 requested a full tablet report, sending...
I20260928 02:36:53.609853 14125 ts_manager.cc:194] Registered new tserver with Master: ffe57bc5a65c4841aac9b43bf3205766 (127.0.98.2:36069)
I20260928 02:36:53.610225 14125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44159
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 02:36:53.628417 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33383
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:36:53.705663 14455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:36:53.705839 14455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:36:53.705868 14455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:36:53.706933 14455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:36:53.706964 14455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:36:53.706987 14455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:36:53.708312 14455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:36:53.708468 14455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:36:53.708632 14455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:36:53.713131 14461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:53.713183 14455 server_base.cc:1034] running on GCE node
W20260928 02:36:53.713131 14463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:53.713186 14460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:53.713519 14455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:36:53.713698 14455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:36:53.714821 14455 hybrid_clock.cc:648] HybridClock initialized: now 1790563013714810 us; error 27 us; skew 500 ppm
Sep 28 02:36:53 dist-test-slave-fz47 krb5kdc[14080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:36:53.717058 14455 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 02:36:53.717490 14455 webserver.cc:460] Webserver started at http://127.0.98.3:39447/ using document root <none> and password file <none>
I20260928 02:36:53.717685 14455 fs_manager.cc:359] Metadata directory not provided
I20260928 02:36:53.717731 14455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:36:53.717846 14455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:36:53.718783 14455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/data/instance:
uuid: "fc893f328a02415b90838e91522ca56b"
format_stamp: "Formatted at 2026-09-28 02:36:53 on dist-test-slave-fz47"
server_key: "7db6c44f0bd4d0ded2b354e4774713f6"
server_key_iv: "7b68eae3cea99d4890d3a6bb6a033c53"
server_key_version: "encryptionkey@0"
I20260928 02:36:53.719100 14455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fc893f328a02415b90838e91522ca56b"
format_stamp: "Formatted at 2026-09-28 02:36:53 on dist-test-slave-fz47"
server_key: "7db6c44f0bd4d0ded2b354e4774713f6"
server_key_iv: "7b68eae3cea99d4890d3a6bb6a033c53"
server_key_version: "encryptionkey@0"
I20260928 02:36:53.720269 14455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:36:53.720871 14470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:53.721012 14455 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:36:53.721088 14455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "fc893f328a02415b90838e91522ca56b"
format_stamp: "Formatted at 2026-09-28 02:36:53 on dist-test-slave-fz47"
server_key: "7db6c44f0bd4d0ded2b354e4774713f6"
server_key_iv: "7b68eae3cea99d4890d3a6bb6a033c53"
server_key_version: "encryptionkey@0"
I20260928 02:36:53.721145 14455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:36:53.735404 14455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:36:53.736908 14455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:36:53.737027 14455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:36:53.737206 14455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:36:53.737498 14455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:36:53.737530 14455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:53.737555 14455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:36:53.737572 14455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:53.742904 14455 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35121
I20260928 02:36:53.742935 14583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35121 every 8 connection(s)
I20260928 02:36:53.743211 14455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 02:36:53 dist-test-slave-fz47 krb5kdc[14080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:36:53.751209 14164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:53.743840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50071 (local address 127.0.98.62:33383)
0928 02:36:53.743884 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:53.743887 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:53.744051 (+ 164us) server_negotiation.cc:407] Connection header received
0928 02:36:53.744556 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:53.744557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:53.744569 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:53.744617 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:53.745552 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:53.746055 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:53.746991 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:53.747179 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:53.747230 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:53.749062 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:53.749064 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:53.749064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:53.749070 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:53.750183 (+ 1113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:53.750507 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:53.750507 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:53.750508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:53.750554 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:53.750749 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:53.750750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:53.750750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:53.750850 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:53.750902 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:53.751096 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 02:36:53.751149 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:36:53.751348 14586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:53.743814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.3:50071)
0928 02:36:53.743941 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:53.743977 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:53.744465 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:53.744691 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:53.744695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:53.745233 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:36:53.745470 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:53.745475 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:53.746131 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:53.746132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:53.746914 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:53.746916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:53.747241 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:53.747242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:53.747348 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:53.747738 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:36:53.747745 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:36:53.748972 (+ 1227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:36:53.750250 (+ 1278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:53.750256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:53.750258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:53.750453 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:53.750611 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:53.750611 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:53.750612 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:53.750715 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:53.750897 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:36:53.750900 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:36:53.751006 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 02:36:53.751076 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 02:36:53.751180 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":63,"threads_started":1}
I20260928 02:36:53.751758 14584 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33383
I20260928 02:36:53.751840 14584 heartbeater.cc:464] Registering TS with master...
I20260928 02:36:53.752007 14584 heartbeater.cc:511] Master 127.0.98.62:33383 requested a full tablet report, sending...
I20260928 02:36:53.752324 14125 ts_manager.cc:194] Registered new tserver with Master: fc893f328a02415b90838e91522ca56b (127.0.98.3:35121)
I20260928 02:36:53.752610 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 14455
I20260928 02:36:53.752722 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/wal/instance
I20260928 02:36:53.752759 14125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50071
I20260928 02:36:53.752883 392 external_mini_cluster.cc:1428] Setting key 579cee6521fefaf4f8997ece5d6d39dc
I20260928 02:36:53.753647 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:36:53.759927 14164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:53.754824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39418 (local address 127.0.98.62:33383)
0928 02:36:53.754895 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:53.754896 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:53.754986 (+ 90us) server_negotiation.cc:407] Connection header received
0928 02:36:53.755043 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:53.755044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:53.755052 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:53.755092 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:53.755584 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:53.756031 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:53.756710 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:53.756955 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:53.757008 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:53.757967 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:53.757968 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:53.757969 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:53.757975 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:53.758993 (+ 1018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:53.759266 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:53.759267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:53.759267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:53.759314 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:53.759469 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:53.759470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:53.759470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:53.759591 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:53.759672 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:53.759774 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 02:36:53.759839 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 02:36:53.761082 14125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:36:53.762660 14125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:36:53.768350 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:53.764473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37061 (local address 127.0.0.1:35668)
0928 02:36:53.764728 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:53.764737 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:53.764806 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:53.765143 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:53.765144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:53.765145 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:36:53.765170 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:53.765171 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:53.766153 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:53.766154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:53.767251 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:53.767253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:53.768079 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:53.768080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:53.768156 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:53.768168 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 02:36:53.768231 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 02:36:53.768281 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":70,"threads_started":1}
I20260928 02:36:53.768366 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:53.764550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51246 (local address 127.0.98.3:35121)
0928 02:36:53.764683 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:53.764687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:53.764793 (+ 106us) server_negotiation.cc:407] Connection header received
0928 02:36:53.764912 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:53.764914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:53.764926 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:53.764999 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:36:53.765147 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:53.766431 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:53.767457 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:53.768045 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:53.768113 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:53.768161 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:53.768261 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 02:36:53.768300 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":69,"threads_started":1}
I20260928 02:36:53.768380 14595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:53.764576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35668 (local address 127.0.98.1:37061)
0928 02:36:53.764754 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:53.764758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:53.764786 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:36:53.764907 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:53.764909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:53.764925 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:53.765000 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:36:53.765296 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:53.766067 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:53.767345 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:53.767964 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:53.767999 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:53.768035 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:53.768259 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 02:36:53.768303 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":73,"threads_started":1}
I20260928 02:36:53.768759 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:53.764647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36069 (local address 127.0.0.1:39914)
0928 02:36:53.764975 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:53.764992 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:53.765033 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:53.765279 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:53.765280 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:53.765281 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:36:53.765308 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:53.765308 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:53.766533 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:53.766534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:53.767639 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:53.767641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:53.768546 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:53.768547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:53.768609 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:53.768614 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:36:53.768657 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 02:36:53.768693 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":28,"thread_start_us":53,"threads_started":1}
I20260928 02:36:53.768795 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:53.764821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39914 (local address 127.0.98.2:36069)
0928 02:36:53.765123 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:53.765128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:53.765137 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:36:53.765170 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:53.765172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:53.765187 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:53.765284 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:36:53.765351 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:53.766447 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:53.767714 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:53.768394 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:53.768456 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:53.768514 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:53.768682 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 02:36:53.768723 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":75,"threads_started":1}
I20260928 02:36:53.769248 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:53.764468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35121 (local address 127.0.0.1:51246)
0928 02:36:53.764698 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:53.764736 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:53.764821 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:53.765021 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:53.765025 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:53.765026 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:36:53.765084 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:53.765086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:53.766543 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:53.766544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:53.767366 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:53.767369 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:53.768120 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:53.768121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:53.768172 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:53.768174 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:36:53.768740 (+ 566us) client_negotiation.cc:240] Negotiation successful
0928 02:36:53.768786 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":29,"thread_start_us":53,"threads_started":1}
I20260928 02:36:53.769132 14250 tablet_service.cc:1467] Processing CreateTablet for tablet 1466fd93ac4146e6bfed8538dc3fb1cf (DEFAULT_TABLE table=test-table [id=65864b2fcec047f4959fd91ff230a17e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:36:53.769495 14250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1466fd93ac4146e6bfed8538dc3fb1cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:36:53.769471 14384 tablet_service.cc:1467] Processing CreateTablet for tablet 1466fd93ac4146e6bfed8538dc3fb1cf (DEFAULT_TABLE table=test-table [id=65864b2fcec047f4959fd91ff230a17e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:36:53.769793 14384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1466fd93ac4146e6bfed8538dc3fb1cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:36:53.770116 14518 tablet_service.cc:1467] Processing CreateTablet for tablet 1466fd93ac4146e6bfed8538dc3fb1cf (DEFAULT_TABLE table=test-table [id=65864b2fcec047f4959fd91ff230a17e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:36:53.770373 14518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1466fd93ac4146e6bfed8538dc3fb1cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:36:53.772086 14598 tablet_bootstrap.cc:492] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766: Bootstrap starting.
I20260928 02:36:53.772719 14599 tablet_bootstrap.cc:492] T 1466fd93ac4146e6bfed8538dc3fb1cf P 589eafa5db924d669a9261c74dce4cb0: Bootstrap starting.
I20260928 02:36:53.772768 14598 tablet_bootstrap.cc:654] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766: Neither blocks nor log segments found. Creating new log.
I20260928 02:36:53.773227 14600 tablet_bootstrap.cc:492] T 1466fd93ac4146e6bfed8538dc3fb1cf P fc893f328a02415b90838e91522ca56b: Bootstrap starting.
I20260928 02:36:53.773331 14599 tablet_bootstrap.cc:654] T 1466fd93ac4146e6bfed8538dc3fb1cf P 589eafa5db924d669a9261c74dce4cb0: Neither blocks nor log segments found. Creating new log.
I20260928 02:36:53.773399 14598 log.cc:824] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766: Log is configured to *not* fsync() on all Append() calls
I20260928 02:36:53.773874 14599 log.cc:824] T 1466fd93ac4146e6bfed8538dc3fb1cf P 589eafa5db924d669a9261c74dce4cb0: Log is configured to *not* fsync() on all Append() calls
I20260928 02:36:53.774006 14600 tablet_bootstrap.cc:654] T 1466fd93ac4146e6bfed8538dc3fb1cf P fc893f328a02415b90838e91522ca56b: Neither blocks nor log segments found. Creating new log.
I20260928 02:36:53.774155 14598 tablet_bootstrap.cc:492] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766: No bootstrap required, opened a new log
I20260928 02:36:53.774224 14598 ts_tablet_manager.cc:1397] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:36:53.774501 14600 log.cc:824] T 1466fd93ac4146e6bfed8538dc3fb1cf P fc893f328a02415b90838e91522ca56b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:36:53.775182 14600 tablet_bootstrap.cc:492] T 1466fd93ac4146e6bfed8538dc3fb1cf P fc893f328a02415b90838e91522ca56b: No bootstrap required, opened a new log
I20260928 02:36:53.775249 14600 ts_tablet_manager.cc:1397] T 1466fd93ac4146e6bfed8538dc3fb1cf P fc893f328a02415b90838e91522ca56b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:36:53.776141 14598 raft_consensus.cc:348] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 } } peers { permanent_uuid: "fc893f328a02415b90838e91522ca56b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35121 } } peers { permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } }
I20260928 02:36:53.776314 14598 raft_consensus.cc:374] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:36:53.776361 14598 raft_consensus.cc:729] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffe57bc5a65c4841aac9b43bf3205766, State: Initialized, Role: FOLLOWER
I20260928 02:36:53.776472 14598 consensus_queue.cc:260] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 } } peers { permanent_uuid: "fc893f328a02415b90838e91522ca56b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35121 } } peers { permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } }
I20260928 02:36:53.776733 14598 ts_tablet_manager.cc:1428] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:36:53.776764 14450 heartbeater.cc:503] Master 127.0.98.62:33383 was elected leader, sending a full tablet report...
I20260928 02:36:53.777062 14600 raft_consensus.cc:348] T 1466fd93ac4146e6bfed8538dc3fb1cf P fc893f328a02415b90838e91522ca56b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 } } peers { permanent_uuid: "fc893f328a02415b90838e91522ca56b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35121 } } peers { permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } }
I20260928 02:36:53.777225 14600 raft_consensus.cc:374] T 1466fd93ac4146e6bfed8538dc3fb1cf P fc893f328a02415b90838e91522ca56b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:36:53.777285 14600 raft_consensus.cc:729] T 1466fd93ac4146e6bfed8538dc3fb1cf P fc893f328a02415b90838e91522ca56b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc893f328a02415b90838e91522ca56b, State: Initialized, Role: FOLLOWER
I20260928 02:36:53.777393 14600 consensus_queue.cc:260] T 1466fd93ac4146e6bfed8538dc3fb1cf P fc893f328a02415b90838e91522ca56b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 } } peers { permanent_uuid: "fc893f328a02415b90838e91522ca56b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35121 } } peers { permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } }
I20260928 02:36:53.777639 14600 ts_tablet_manager.cc:1428] T 1466fd93ac4146e6bfed8538dc3fb1cf P fc893f328a02415b90838e91522ca56b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:36:53.777746 14584 heartbeater.cc:503] Master 127.0.98.62:33383 was elected leader, sending a full tablet report...
I20260928 02:36:53.777848 14599 tablet_bootstrap.cc:492] T 1466fd93ac4146e6bfed8538dc3fb1cf P 589eafa5db924d669a9261c74dce4cb0: No bootstrap required, opened a new log
I20260928 02:36:53.777911 14599 ts_tablet_manager.cc:1397] T 1466fd93ac4146e6bfed8538dc3fb1cf P 589eafa5db924d669a9261c74dce4cb0: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260928 02:36:53.779227 14599 raft_consensus.cc:348] T 1466fd93ac4146e6bfed8538dc3fb1cf P 589eafa5db924d669a9261c74dce4cb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 } } peers { permanent_uuid: "fc893f328a02415b90838e91522ca56b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35121 } } peers { permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } }
I20260928 02:36:53.779346 14599 raft_consensus.cc:374] T 1466fd93ac4146e6bfed8538dc3fb1cf P 589eafa5db924d669a9261c74dce4cb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:36:53.779366 14599 raft_consensus.cc:729] T 1466fd93ac4146e6bfed8538dc3fb1cf P 589eafa5db924d669a9261c74dce4cb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 589eafa5db924d669a9261c74dce4cb0, State: Initialized, Role: FOLLOWER
I20260928 02:36:53.779461 14599 consensus_queue.cc:260] T 1466fd93ac4146e6bfed8538dc3fb1cf P 589eafa5db924d669a9261c74dce4cb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 } } peers { permanent_uuid: "fc893f328a02415b90838e91522ca56b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35121 } } peers { permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } }
I20260928 02:36:53.779682 14599 ts_tablet_manager.cc:1428] T 1466fd93ac4146e6bfed8538dc3fb1cf P 589eafa5db924d669a9261c74dce4cb0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 02:36:53.779716 14316 heartbeater.cc:503] Master 127.0.98.62:33383 was elected leader, sending a full tablet report...
W20260928 02:36:53.849910 14451 tablet.cc:2369] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:36:53.964454 14317 tablet.cc:2369] T 1466fd93ac4146e6bfed8538dc3fb1cf P 589eafa5db924d669a9261c74dce4cb0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:36:53.993721 14585 tablet.cc:2369] T 1466fd93ac4146e6bfed8538dc3fb1cf P fc893f328a02415b90838e91522ca56b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:36:54.061941 14604 raft_consensus.cc:482] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:36:54.062072 14604 raft_consensus.cc:504] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 } } peers { permanent_uuid: "fc893f328a02415b90838e91522ca56b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35121 } } peers { permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } }
I20260928 02:36:54.062438 14604 leader_election.cc:288] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 589eafa5db924d669a9261c74dce4cb0 (127.0.98.1:37061), fc893f328a02415b90838e91522ca56b (127.0.98.3:35121)
I20260928 02:36:54.066104 14595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:54.062697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46341 (local address 127.0.98.1:37061)
0928 02:36:54.062741 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:54.062743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:54.062767 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:36:54.062797 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:54.062798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:54.062810 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:54.062884 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:36:54.063037 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.063787 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.065164 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.065753 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.065822 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:54.065833 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:54.065995 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 02:36:54.066032 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 02:36:54.066104 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:54.062706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49147 (local address 127.0.98.3:35121)
0928 02:36:54.062807 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:54.062809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:54.062819 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:36:54.062870 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:54.062871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:54.062881 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:54.062933 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:36:54.063062 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.063832 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.065174 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.065753 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.065824 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:54.065833 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:54.066005 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 02:36:54.066039 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 02:36:54.066118 14452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:54.062609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37061 (local address 127.0.98.2:46341)
0928 02:36:54.062669 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:54.062680 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:54.062722 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:54.062896 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:54.062896 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:54.062898 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:36:54.062974 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:54.062976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.063860 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.063861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:54.065075 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:54.065079 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.065833 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.065834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:54.065930 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:54.065933 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:36:54.065984 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 02:36:54.066026 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":8}
I20260928 02:36:54.066123 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:54.062609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35121 (local address 127.0.98.2:49147)
0928 02:36:54.062780 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:54.062788 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:54.062823 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:54.062940 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:54.062941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:54.062942 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:36:54.063005 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:54.063006 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.063897 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.063898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:54.065075 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:54.065079 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.065833 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.065834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:54.065930 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:54.065933 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:36:54.065990 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 02:36:54.066025 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":38,"thread_start_us":89,"threads_started":1}
I20260928 02:36:54.066502 14270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1466fd93ac4146e6bfed8538dc3fb1cf" candidate_uuid: "ffe57bc5a65c4841aac9b43bf3205766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "589eafa5db924d669a9261c74dce4cb0" is_pre_election: true
I20260928 02:36:54.066644 14270 raft_consensus.cc:2454] T 1466fd93ac4146e6bfed8538dc3fb1cf P 589eafa5db924d669a9261c74dce4cb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffe57bc5a65c4841aac9b43bf3205766 in term 0.
I20260928 02:36:54.066699 14538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1466fd93ac4146e6bfed8538dc3fb1cf" candidate_uuid: "ffe57bc5a65c4841aac9b43bf3205766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc893f328a02415b90838e91522ca56b" is_pre_election: true
I20260928 02:36:54.066848 14538 raft_consensus.cc:2454] T 1466fd93ac4146e6bfed8538dc3fb1cf P fc893f328a02415b90838e91522ca56b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffe57bc5a65c4841aac9b43bf3205766 in term 0.
I20260928 02:36:54.066879 14337 leader_election.cc:302] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 589eafa5db924d669a9261c74dce4cb0, ffe57bc5a65c4841aac9b43bf3205766; no voters:
I20260928 02:36:54.067101 14604 raft_consensus.cc:2781] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:36:54.067153 14604 raft_consensus.cc:482] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:36:54.067168 14604 raft_consensus.cc:3037] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:36:54.067803 14604 raft_consensus.cc:504] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 } } peers { permanent_uuid: "fc893f328a02415b90838e91522ca56b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35121 } } peers { permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } }
I20260928 02:36:54.067914 14604 leader_election.cc:288] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766 [CANDIDATE]: Term 1 election: Requested vote from peers 589eafa5db924d669a9261c74dce4cb0 (127.0.98.1:37061), fc893f328a02415b90838e91522ca56b (127.0.98.3:35121)
I20260928 02:36:54.068071 14538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1466fd93ac4146e6bfed8538dc3fb1cf" candidate_uuid: "ffe57bc5a65c4841aac9b43bf3205766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc893f328a02415b90838e91522ca56b"
I20260928 02:36:54.068101 14270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1466fd93ac4146e6bfed8538dc3fb1cf" candidate_uuid: "ffe57bc5a65c4841aac9b43bf3205766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "589eafa5db924d669a9261c74dce4cb0"
I20260928 02:36:54.068142 14538 raft_consensus.cc:3037] T 1466fd93ac4146e6bfed8538dc3fb1cf P fc893f328a02415b90838e91522ca56b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:36:54.068159 14270 raft_consensus.cc:3037] T 1466fd93ac4146e6bfed8538dc3fb1cf P 589eafa5db924d669a9261c74dce4cb0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:36:54.068812 14538 raft_consensus.cc:2454] T 1466fd93ac4146e6bfed8538dc3fb1cf P fc893f328a02415b90838e91522ca56b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffe57bc5a65c4841aac9b43bf3205766 in term 1.
I20260928 02:36:54.068883 14270 raft_consensus.cc:2454] T 1466fd93ac4146e6bfed8538dc3fb1cf P 589eafa5db924d669a9261c74dce4cb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffe57bc5a65c4841aac9b43bf3205766 in term 1.
I20260928 02:36:54.068985 14338 leader_election.cc:302] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fc893f328a02415b90838e91522ca56b, ffe57bc5a65c4841aac9b43bf3205766; no voters:
I20260928 02:36:54.069072 14604 raft_consensus.cc:2781] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:36:54.069226 14604 raft_consensus.cc:686] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766 [term 1 LEADER]: Becoming Leader. State: Replica: ffe57bc5a65c4841aac9b43bf3205766, State: Running, Role: LEADER
I20260928 02:36:54.069360 14604 consensus_queue.cc:237] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 } } peers { permanent_uuid: "fc893f328a02415b90838e91522ca56b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35121 } } peers { permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } }
I20260928 02:36:54.070144 14125 catalog_manager.cc:5495] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766 reported cstate change: term changed from 0 to 1, leader changed from <none> to ffe57bc5a65c4841aac9b43bf3205766 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ffe57bc5a65c4841aac9b43bf3205766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc893f328a02415b90838e91522ca56b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } health_report { overall_health: HEALTHY } } }
I20260928 02:36:54.120185 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:54.118062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39926 (local address 127.0.98.2:36069)
0928 02:36:54.118110 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:54.118112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:54.118122 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:36:54.118161 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:54.118162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:54.118180 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:54.118237 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:36:54.118361 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.118856 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.119576 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.119762 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.119806 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:54.119950 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:36:54.120041 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:36:54.120098 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:54.120110 (+ 12us) server_negotiation.cc:299] Negotiation successful
0928 02:36:54.120126 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 02:36:54.121914 14538 raft_consensus.cc:1261] T 1466fd93ac4146e6bfed8538dc3fb1cf P fc893f328a02415b90838e91522ca56b [term 1 FOLLOWER]: Refusing update from remote peer ffe57bc5a65c4841aac9b43bf3205766: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:36:54.122004 14270 raft_consensus.cc:1261] T 1466fd93ac4146e6bfed8538dc3fb1cf P 589eafa5db924d669a9261c74dce4cb0 [term 1 FOLLOWER]: Refusing update from remote peer ffe57bc5a65c4841aac9b43bf3205766: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:36:54.122220 14608 consensus_queue.cc:1035] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc893f328a02415b90838e91522ca56b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:36:54.122359 14604 consensus_queue.cc:1035] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:36:54.124661 14613 mvcc.cc:205] Tried to move back new op lower bound from 7334146105840926720 to 7334146105628827648. Current Snapshot: MvccSnapshot[applied={T|T < 7334146105840926720}]
I20260928 02:36:54.125229 14612 mvcc.cc:205] Tried to move back new op lower bound from 7334146105840926720 to 7334146105628827648. Current Snapshot: MvccSnapshot[applied={T|T < 7334146105840926720}]
I20260928 02:36:54.126585 14124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:36:54.126680 14124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:36:54.127398 14124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13: Sending DeleteTablet for 3 replicas of tablet 1466fd93ac4146e6bfed8538dc3fb1cf
I20260928 02:36:54.127735 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 14187
I20260928 02:36:54.127735 14518 tablet_service.cc:1514] Processing DeleteTablet for tablet 1466fd93ac4146e6bfed8538dc3fb1cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:36:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51246
I20260928 02:36:54.127735 14250 tablet_service.cc:1514] Processing DeleteTablet for tablet 1466fd93ac4146e6bfed8538dc3fb1cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:36:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35668
I20260928 02:36:54.127735 14384 tablet_service.cc:1514] Processing DeleteTablet for tablet 1466fd93ac4146e6bfed8538dc3fb1cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:36:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39914
I20260928 02:36:54.128290 14621 tablet_replica.cc:330] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766: stopping tablet replica
I20260928 02:36:54.128382 14621 raft_consensus.cc:2229] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:36:54.128458 14621 raft_consensus.cc:2258] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:36:54.128580 14622 tablet_replica.cc:330] T 1466fd93ac4146e6bfed8538dc3fb1cf P fc893f328a02415b90838e91522ca56b: stopping tablet replica
I20260928 02:36:54.128654 14622 raft_consensus.cc:2229] T 1466fd93ac4146e6bfed8538dc3fb1cf P fc893f328a02415b90838e91522ca56b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:36:54.128726 14622 raft_consensus.cc:2258] T 1466fd93ac4146e6bfed8538dc3fb1cf P fc893f328a02415b90838e91522ca56b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:36:54.129004 14621 ts_tablet_manager.cc:1905] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:36:54.129065 14622 ts_tablet_manager.cc:1905] T 1466fd93ac4146e6bfed8538dc3fb1cf P fc893f328a02415b90838e91522ca56b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:36:54.130661 14621 ts_tablet_manager.cc:1918] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:36:54.130662 14622 ts_tablet_manager.cc:1918] T 1466fd93ac4146e6bfed8538dc3fb1cf P fc893f328a02415b90838e91522ca56b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:36:54.130725 14621 log.cc:1196] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/wal/wals/1466fd93ac4146e6bfed8538dc3fb1cf
I20260928 02:36:54.130868 14622 log.cc:1196] T 1466fd93ac4146e6bfed8538dc3fb1cf P fc893f328a02415b90838e91522ca56b: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/wal/wals/1466fd93ac4146e6bfed8538dc3fb1cf
I20260928 02:36:54.130963 14621 ts_tablet_manager.cc:1939] T 1466fd93ac4146e6bfed8538dc3fb1cf P ffe57bc5a65c4841aac9b43bf3205766: Deleting consensus metadata
I20260928 02:36:54.131163 14622 ts_tablet_manager.cc:1939] T 1466fd93ac4146e6bfed8538dc3fb1cf P fc893f328a02415b90838e91522ca56b: Deleting consensus metadata
I20260928 02:36:54.131376 14110 catalog_manager.cc:4841] TS ffe57bc5a65c4841aac9b43bf3205766 (127.0.98.2:36069): tablet 1466fd93ac4146e6bfed8538dc3fb1cf (table test-table [id=65864b2fcec047f4959fd91ff230a17e]) successfully deleted
I20260928 02:36:54.131688 14110 catalog_manager.cc:4841] TS fc893f328a02415b90838e91522ca56b (127.0.98.3:35121): tablet 1466fd93ac4146e6bfed8538dc3fb1cf (table test-table [id=65864b2fcec047f4959fd91ff230a17e]) successfully deleted
W20260928 02:36:54.133747 14109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37061 (error 108)
I20260928 02:36:54.134189 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:54.134071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37061 (local address 127.0.0.1:35684)
0928 02:36:54.134121 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:54.134157 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 02:36:54.134197 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 14321
W20260928 02:36:54.136197 14109 catalog_manager.cc:4584] TS 589eafa5db924d669a9261c74dce4cb0 (127.0.98.1:37061): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1466fd93ac4146e6bfed8538dc3fb1cf: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37061: connect: Connection refused (error 111)
I20260928 02:36:54.138935 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 14455
I20260928 02:36:54.142297 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 14093
I20260928 02:36:54.146476 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33383
--webserver_interface=127.0.98.62
--webserver_port=43781
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:36:54.223968 14624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:36:54.224162 14624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:36:54.224190 14624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:36:54.225315 14624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:36:54.225350 14624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:36:54.225368 14624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:36:54.225383 14624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:36:54.225399 14624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:36:54.226826 14624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:36:54.227059 14624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:36:54.227257 14624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:36:54.235090 14630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:54.235255 14624 server_base.cc:1034] running on GCE node
W20260928 02:36:54.235081 14633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:54.235081 14631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:54.235621 14624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:36:54.235908 14624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:36:54.237041 14624 hybrid_clock.cc:648] HybridClock initialized: now 1790563014237027 us; error 32 us; skew 500 ppm
Sep 28 02:36:54 dist-test-slave-fz47 krb5kdc[14080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:36:54.239310 14624 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:36:54.239717 14624 webserver.cc:460] Webserver started at http://127.0.98.62:43781/ using document root <none> and password file <none>
I20260928 02:36:54.239953 14624 fs_manager.cc:359] Metadata directory not provided
I20260928 02:36:54.240015 14624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:36:54.241274 14624 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:36:54.241832 14640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:54.241986 14624 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:54.242065 14624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "34384d463a834c0ea4abe376446aab13"
format_stamp: "Formatted at 2026-09-28 02:36:53 on dist-test-slave-fz47"
server_key: "a502da60ccceaebb557b50273b4101b8"
server_key_iv: "b9cc61d58beb3be4c243215166552638"
server_key_version: "encryptionkey@0"
I20260928 02:36:54.242424 14624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:36:54.258651 14624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:36:54.260474 14624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:36:54.260607 14624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:36:54.264539 14624 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33383
I20260928 02:36:54.264571 14692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33383 every 8 connection(s)
I20260928 02:36:54.264947 14624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:36:54.266484 14693 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:36:54.267125 14693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13: Bootstrap starting.
I20260928 02:36:54.270449 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 14624
I20260928 02:36:54.270712 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37061
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33383
--webserver_port=44669
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:36:54.274884 14693 log.cc:824] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13: Log is configured to *not* fsync() on all Append() calls
I20260928 02:36:54.276661 14693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:36:54.276889 14693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13: Bootstrap complete.
I20260928 02:36:54.278987 14693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34384d463a834c0ea4abe376446aab13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33383 } }
I20260928 02:36:54.279209 14693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34384d463a834c0ea4abe376446aab13, State: Initialized, Role: FOLLOWER
I20260928 02:36:54.279351 14693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34384d463a834c0ea4abe376446aab13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33383 } }
I20260928 02:36:54.279420 14693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:36:54.279474 14693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:36:54.279520 14693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:36:54.280459 14693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34384d463a834c0ea4abe376446aab13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33383 } }
I20260928 02:36:54.280547 14693 leader_election.cc:302] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34384d463a834c0ea4abe376446aab13; no voters:
I20260928 02:36:54.280691 14693 leader_election.cc:288] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:36:54.280748 14697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:36:54.280926 14693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:36:54.280937 14697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [term 2 LEADER]: Becoming Leader. State: Replica: 34384d463a834c0ea4abe376446aab13, State: Running, Role: LEADER
I20260928 02:36:54.281095 14697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34384d463a834c0ea4abe376446aab13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33383 } }
I20260928 02:36:54.281366 14697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34384d463a834c0ea4abe376446aab13. Latest consensus state: current_term: 2 leader_uuid: "34384d463a834c0ea4abe376446aab13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34384d463a834c0ea4abe376446aab13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33383 } } }
I20260928 02:36:54.281427 14697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [sys.catalog]: This master's current role is: LEADER
I20260928 02:36:54.281915 14698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "34384d463a834c0ea4abe376446aab13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34384d463a834c0ea4abe376446aab13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33383 } } }
I20260928 02:36:54.281991 14698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [sys.catalog]: This master's current role is: LEADER
I20260928 02:36:54.282099 14704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:36:54.282895 14704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:36:54.283010 14704 catalog_manager.cc:1259] Loaded cluster ID: 7ab64556213d428eaaf93afd40278b18
I20260928 02:36:54.283041 14704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:36:54.283253 14704 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 02:36:54.288205 14704 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 02:36:54.290477 14704 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:36:54.290657 14704 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13: Loaded TSK: 0
I20260928 02:36:54.290861 14704 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:36:54.351686 14695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:36:54.351869 14695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:36:54.351898 14695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:36:54.352993 14695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:36:54.353029 14695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:36:54.353055 14695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:36:54.354511 14695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37061
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44669
--tserver_master_addrs=127.0.98.62:33383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:36:54.354718 14695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:36:54.354954 14695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:36:54.359467 14721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:54.359521 14720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:54.359625 14695 server_base.cc:1034] running on GCE node
W20260928 02:36:54.359788 14723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:54.359992 14695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:36:54.360257 14695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:36:54.361423 14695 hybrid_clock.cc:648] HybridClock initialized: now 1790563014361406 us; error 30 us; skew 500 ppm
Sep 28 02:36:54 dist-test-slave-fz47 krb5kdc[14080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:36:54.364187 14695 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 02:36:54.364667 14695 webserver.cc:460] Webserver started at http://127.0.98.1:44669/ using document root <none> and password file <none>
I20260928 02:36:54.364888 14695 fs_manager.cc:359] Metadata directory not provided
I20260928 02:36:54.364969 14695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:36:54.366556 14695 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:36:54.367280 14730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:54.367432 14695 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:36:54.367501 14695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "589eafa5db924d669a9261c74dce4cb0"
format_stamp: "Formatted at 2026-09-28 02:36:53 on dist-test-slave-fz47"
server_key: "dd33b1baca925e99708089302a005fae"
server_key_iv: "39c70270a11f9d4ef3fdac9f24aa809c"
server_key_version: "encryptionkey@0"
I20260928 02:36:54.367822 14695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:36:54.382949 14695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:36:54.384871 14695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:36:54.384977 14695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:36:54.385180 14695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:36:54.385594 14737 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 02:36:54.387346 14695 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:36:54.387395 14695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 02:36:54.387429 14695 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:36:54.388015 14695 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:36:54.388046 14695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 02:36:54.388116 14737 tablet_bootstrap.cc:492] T 1466fd93ac4146e6bfed8538dc3fb1cf P 589eafa5db924d669a9261c74dce4cb0: Bootstrap starting.
I20260928 02:36:54.394284 14695 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37061
I20260928 02:36:54.394620 14695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
I20260928 02:36:54.395226 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 14695
I20260928 02:36:54.395506 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36069
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33383
--webserver_port=36877
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:36:54.398411 14737 log.cc:824] T 1466fd93ac4146e6bfed8538dc3fb1cf P 589eafa5db924d669a9261c74dce4cb0: Log is configured to *not* fsync() on all Append() calls
I20260928 02:36:54.399952 14844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37061 every 8 connection(s)
I20260928 02:36:54.400730 14737 tablet_bootstrap.cc:492] T 1466fd93ac4146e6bfed8538dc3fb1cf P 589eafa5db924d669a9261c74dce4cb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:36:54.401000 14737 tablet_bootstrap.cc:492] T 1466fd93ac4146e6bfed8538dc3fb1cf P 589eafa5db924d669a9261c74dce4cb0: Bootstrap complete.
I20260928 02:36:54.402083 14737 ts_tablet_manager.cc:1397] T 1466fd93ac4146e6bfed8538dc3fb1cf P 589eafa5db924d669a9261c74dce4cb0: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260928 02:36:54.403435 14737 raft_consensus.cc:348] T 1466fd93ac4146e6bfed8538dc3fb1cf P 589eafa5db924d669a9261c74dce4cb0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 } } peers { permanent_uuid: "fc893f328a02415b90838e91522ca56b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35121 } } peers { permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } }
I20260928 02:36:54.403743 14737 raft_consensus.cc:729] T 1466fd93ac4146e6bfed8538dc3fb1cf P 589eafa5db924d669a9261c74dce4cb0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 589eafa5db924d669a9261c74dce4cb0, State: Initialized, Role: FOLLOWER
I20260928 02:36:54.403877 14737 consensus_queue.cc:260] T 1466fd93ac4146e6bfed8538dc3fb1cf P 589eafa5db924d669a9261c74dce4cb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 } } peers { permanent_uuid: "fc893f328a02415b90838e91522ca56b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35121 } } peers { permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } }
I20260928 02:36:54.404114 14737 ts_tablet_manager.cc:1428] T 1466fd93ac4146e6bfed8538dc3fb1cf P 589eafa5db924d669a9261c74dce4cb0: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
Sep 28 02:36:54 dist-test-slave-fz47 krb5kdc[14080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:36:54.409392 14848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:54.395315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50523 (local address 127.0.98.62:33383)
0928 02:36:54.398701 (+ 3386us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:54.398713 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:54.398993 (+ 280us) server_negotiation.cc:407] Connection header received
0928 02:36:54.399701 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:54.399708 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:54.399775 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:54.399916 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:54.400911 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.402252 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.403670 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.404275 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.404381 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:54.406393 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:54.406398 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:54.406398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:54.406409 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:54.407844 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:54.408204 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:54.408207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:54.408208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:54.408293 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:54.408505 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:54.408506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:54.408507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:54.408713 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:54.408789 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:54.409059 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 02:36:54.409177 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3330,"thread_start_us":74,"threads_started":1}
I20260928 02:36:54.409394 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:54.395239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.1:50523)
0928 02:36:54.398803 (+ 3564us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:54.398853 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:54.399479 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:54.400119 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:54.400123 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:54.400469 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:36:54.400721 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:54.400726 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.402354 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.402355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:54.403578 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:54.403581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.404355 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.404356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:54.404476 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:54.404965 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:36:54.404973 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:36:54.406302 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:36:54.407920 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:54.407926 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:54.407928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:54.408134 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:54.408353 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:54.408353 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:54.408354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:54.408453 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:54.408786 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:36:54.408793 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:36:54.408919 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 02:36:54.409001 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 02:36:54.409162 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3522,"thread_start_us":91,"threads_started":1}
I20260928 02:36:54.409998 14845 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33383
I20260928 02:36:54.410079 14845 heartbeater.cc:464] Registering TS with master...
I20260928 02:36:54.410233 14845 heartbeater.cc:511] Master 127.0.98.62:33383 requested a full tablet report, sending...
I20260928 02:36:54.410852 14657 ts_manager.cc:194] Registered new tserver with Master: 589eafa5db924d669a9261c74dce4cb0 (127.0.98.1:37061)
I20260928 02:36:54.411849 14657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50523
I20260928 02:36:54.412802 14845 heartbeater.cc:503] Master 127.0.98.62:33383 was elected leader, sending a full tablet report...
Sep 28 02:36:54 dist-test-slave-fz47 krb5kdc[14080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 02:36:54.418942 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:54.411366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37061 (local address 127.0.0.1:35696)
0928 02:36:54.411517 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:54.411556 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:54.411621 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:54.411880 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:54.411884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:54.412168 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:36:54.412208 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:54.412210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.413509 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.413510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:54.414096 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:54.414098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.414669 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.414671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:54.414724 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:54.415143 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:36:54.415148 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:36:54.416299 (+ 1151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:36:54.418027 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:54.418033 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:54.418034 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:54.418205 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:54.418377 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:54.418378 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:54.418378 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:54.418423 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:54.418667 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:36:54.418670 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:36:54.418706 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 02:36:54.418793 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 02:36:54.418870 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20260928 02:36:54.418998 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:54.411581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35696 (local address 127.0.98.1:37061)
0928 02:36:54.411743 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:54.411747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:54.411756 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:36:54.411784 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:54.411786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:54.411800 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:54.411858 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:54.412282 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.413443 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.414167 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.414604 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.414665 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:54.416374 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:54.416381 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:54.416381 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:54.416387 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:54.417960 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:54.418268 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:54.418271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:54.418271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:54.418327 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:54.418468 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:54.418468 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:54.418469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:54.418616 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:54.418687 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:54.418824 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 02:36:54.418909 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":68,"threads_started":1}
I20260928 02:36:54.419354 14777 tablet_service.cc:1514] Processing DeleteTablet for tablet 1466fd93ac4146e6bfed8538dc3fb1cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:36:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35696
I20260928 02:36:54.419903 14854 tablet_replica.cc:330] stopping tablet replica
I20260928 02:36:54.419989 14854 raft_consensus.cc:2229] T 1466fd93ac4146e6bfed8538dc3fb1cf P 589eafa5db924d669a9261c74dce4cb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:36:54.420051 14854 raft_consensus.cc:2258] T 1466fd93ac4146e6bfed8538dc3fb1cf P 589eafa5db924d669a9261c74dce4cb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:36:54.420329 14854 ts_tablet_manager.cc:1905] T 1466fd93ac4146e6bfed8538dc3fb1cf P 589eafa5db924d669a9261c74dce4cb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:36:54.422144 14854 ts_tablet_manager.cc:1918] T 1466fd93ac4146e6bfed8538dc3fb1cf P 589eafa5db924d669a9261c74dce4cb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:36:54.422199 14854 log.cc:1196] T 1466fd93ac4146e6bfed8538dc3fb1cf P 589eafa5db924d669a9261c74dce4cb0: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/wal/wals/1466fd93ac4146e6bfed8538dc3fb1cf
I20260928 02:36:54.422374 14854 ts_tablet_manager.cc:1939] T 1466fd93ac4146e6bfed8538dc3fb1cf P 589eafa5db924d669a9261c74dce4cb0: Deleting consensus metadata
I20260928 02:36:54.422784 14641 catalog_manager.cc:4841] TS 589eafa5db924d669a9261c74dce4cb0 (127.0.98.1:37061): tablet 1466fd93ac4146e6bfed8538dc3fb1cf (table test-table [id=65864b2fcec047f4959fd91ff230a17e]) successfully deleted
W20260928 02:36:54.480732 14849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:36:54.480950 14849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:36:54.480975 14849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:36:54.482208 14849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:36:54.482244 14849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:36:54.482270 14849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:36:54.483678 14849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36069
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36877
--tserver_master_addrs=127.0.98.62:33383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:36:54.483882 14849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:36:54.484084 14849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:36:54.488571 14862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:54.488624 14859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:54.488605 14860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:54.488713 14849 server_base.cc:1034] running on GCE node
I20260928 02:36:54.488890 14849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:36:54.489099 14849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:36:54.490231 14849 hybrid_clock.cc:648] HybridClock initialized: now 1790563014490209 us; error 36 us; skew 500 ppm
Sep 28 02:36:54 dist-test-slave-fz47 krb5kdc[14080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:36:54.492472 14849 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 02:36:54.492853 14849 webserver.cc:460] Webserver started at http://127.0.98.2:36877/ using document root <none> and password file <none>
I20260928 02:36:54.493038 14849 fs_manager.cc:359] Metadata directory not provided
I20260928 02:36:54.493093 14849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:36:54.494390 14849 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:36:54.494850 14869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:54.494992 14849 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:54.495097 14849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "ffe57bc5a65c4841aac9b43bf3205766"
format_stamp: "Formatted at 2026-09-28 02:36:53 on dist-test-slave-fz47"
server_key: "237f17fa9fef4f79568b239884002003"
server_key_iv: "b3de135f1dc25a5092746ea5ef887817"
server_key_version: "encryptionkey@0"
I20260928 02:36:54.495402 14849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:36:54.507107 14849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:36:54.508724 14849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:36:54.508854 14849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:36:54.509045 14849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:36:54.509354 14849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:36:54.509390 14849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:54.509418 14849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:36:54.509438 14849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:54.514945 14849 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36069
I20260928 02:36:54.514984 14982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36069 every 8 connection(s)
I20260928 02:36:54.515286 14849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 02:36:54 dist-test-slave-fz47 krb5kdc[14080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:36:54.523202 14848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:54.515910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58125 (local address 127.0.98.62:33383)
0928 02:36:54.515960 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:54.515962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:54.516159 (+ 197us) server_negotiation.cc:407] Connection header received
0928 02:36:54.516614 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:54.516615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:54.516627 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:54.516668 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:54.517563 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.518043 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.518943 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.519133 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.519178 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:54.520984 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:54.520986 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:54.520986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:54.520992 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:54.522184 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:54.522537 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:54.522538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:54.522538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:54.522579 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:54.522757 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:54.522757 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:54.522758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:54.522843 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:54.522887 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:54.523082 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 02:36:54.523140 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 02:36:54.523379 14985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:54.515905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.2:58125)
0928 02:36:54.516048 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:54.516081 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:54.516539 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:54.516726 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:54.516729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:54.517292 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:36:54.517496 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:54.517501 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.518106 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.518106 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:54.518873 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:54.518875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.519183 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.519184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:54.519271 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:54.519670 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:36:54.519678 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:36:54.520908 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:36:54.522253 (+ 1345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:54.522258 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:54.522260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:54.522475 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:54.522622 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:54.522622 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:54.522622 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:54.522706 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:54.522886 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:36:54.522889 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:36:54.522992 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 02:36:54.523067 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 02:36:54.523174 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260928 02:36:54.523780 14983 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33383
I20260928 02:36:54.523878 14983 heartbeater.cc:464] Registering TS with master...
I20260928 02:36:54.524042 14983 heartbeater.cc:511] Master 127.0.98.62:33383 requested a full tablet report, sending...
I20260928 02:36:54.524134 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 14849
I20260928 02:36:54.524403 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35121
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33383
--webserver_port=39447
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:36:54.524400 14657 ts_manager.cc:194] Registered new tserver with Master: ffe57bc5a65c4841aac9b43bf3205766 (127.0.98.2:36069)
I20260928 02:36:54.524752 14657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58125
W20260928 02:36:54.602070 14986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:36:54.602275 14986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:36:54.602308 14986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:36:54.603474 14986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:36:54.603510 14986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:36:54.603535 14986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:36:54.604913 14986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35121
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39447
--tserver_master_addrs=127.0.98.62:33383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:36:54.605103 14986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:36:54.605314 14986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:36:54.609926 14986 server_base.cc:1034] running on GCE node
W20260928 02:36:54.609899 14992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:54.609890 14991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:54.609899 14994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:54.610235 14986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:36:54.610447 14986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:36:54.611604 14986 hybrid_clock.cc:648] HybridClock initialized: now 1790563014611590 us; error 23 us; skew 500 ppm
Sep 28 02:36:54 dist-test-slave-fz47 krb5kdc[14080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:36:54.613873 14986 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 02:36:54.614260 14986 webserver.cc:460] Webserver started at http://127.0.98.3:39447/ using document root <none> and password file <none>
I20260928 02:36:54.614445 14986 fs_manager.cc:359] Metadata directory not provided
I20260928 02:36:54.614493 14986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:36:54.615656 14986 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:36:54.616163 15001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:54.616303 14986 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:54.616379 14986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "fc893f328a02415b90838e91522ca56b"
format_stamp: "Formatted at 2026-09-28 02:36:53 on dist-test-slave-fz47"
server_key: "7db6c44f0bd4d0ded2b354e4774713f6"
server_key_iv: "7b68eae3cea99d4890d3a6bb6a033c53"
server_key_version: "encryptionkey@0"
I20260928 02:36:54.616691 14986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:36:54.637308 14986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:36:54.639015 14986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:36:54.639127 14986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:36:54.639343 14986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:36:54.639631 14986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:36:54.639663 14986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:54.639691 14986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:36:54.639708 14986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:54.645116 14986 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35121
I20260928 02:36:54.645160 15114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35121 every 8 connection(s)
I20260928 02:36:54.645529 14986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
I20260928 02:36:54.648859 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 14986
Sep 28 02:36:54 dist-test-slave-fz47 krb5kdc[14080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563014, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:36:54.654816 14848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:54.646260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60393 (local address 127.0.98.62:33383)
0928 02:36:54.646319 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:54.646321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:54.646525 (+ 204us) server_negotiation.cc:407] Connection header received
0928 02:36:54.647448 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:54.647449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:54.647460 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:54.647512 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:54.648091 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.648576 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.649728 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.649921 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.649972 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:54.652303 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:54.652304 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:54.652305 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:54.652311 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:54.653494 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:54.653830 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:54.653831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:54.653832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:54.653878 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:54.654129 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:54.654130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:54.654130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:54.654236 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:54.654285 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:54.654646 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 02:36:54.654726 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 02:36:54.654903 15117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:54.646266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.3:60393)
0928 02:36:54.646419 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:54.646457 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:54.647348 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:54.647569 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:54.647572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:54.647805 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:36:54.648000 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:54.648005 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.648640 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.648641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:54.649656 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:54.649659 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.649992 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.649993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:54.650127 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:54.650599 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:36:54.650607 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:36:54.652224 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:36:54.653556 (+ 1332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:54.653563 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:54.653565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:54.653770 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:54.653937 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:54.653938 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:54.653938 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:54.654058 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:54.654293 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:36:54.654299 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:36:54.654476 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 02:36:54.654591 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 02:36:54.654736 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
I20260928 02:36:54.655361 15115 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33383
I20260928 02:36:54.655448 15115 heartbeater.cc:464] Registering TS with master...
I20260928 02:36:54.655611 15115 heartbeater.cc:511] Master 127.0.98.62:33383 requested a full tablet report, sending...
I20260928 02:36:54.655869 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:54.649229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39432 (local address 127.0.98.62:33383)
0928 02:36:54.649470 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:54.649471 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:54.649477 (+ 6us) server_negotiation.cc:407] Connection header received
0928 02:36:54.649493 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:54.649493 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:54.649500 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:54.649544 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:54.650118 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.650599 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.651502 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.651891 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.651957 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:54.653186 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:54.653188 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:54.653188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:54.653195 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:54.654577 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:54.655133 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:54.655134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:54.655135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:54.655193 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:54.655408 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:54.655408 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:54.655409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:54.655527 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:54.655681 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:54.655714 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 02:36:54.655783 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":68,"threads_started":1}
I20260928 02:36:54.655969 14657 ts_manager.cc:194] Registered new tserver with Master: fc893f328a02415b90838e91522ca56b (127.0.98.3:35121)
I20260928 02:36:54.656358 14657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60393
I20260928 02:36:54.656517 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:36:54.659025 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:54.656820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39446 (local address 127.0.98.62:33383)
0928 02:36:54.656857 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:54.656859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:54.657151 (+ 292us) server_negotiation.cc:407] Connection header received
0928 02:36:54.657193 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:54.657194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:54.657219 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:54.657298 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:36:54.657380 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.657835 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.658490 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.658669 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.658715 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:54.658808 (+ 93us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:36:54.658888 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:36:54.658940 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:54.658967 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 02:36:54.658982 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:36:54.659245 14657 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:39446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:36:54.660707 14657 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:36:54.662532 14777 tablet_service.cc:1467] Processing CreateTablet for tablet ebcbe086af124b8180ebc564f31ff7c9 (DEFAULT_TABLE table=test-table [id=cd50d5284def47aab77d3606c18ea215]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:36:54.662752 14777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebcbe086af124b8180ebc564f31ff7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:36:54.664021 14737 tablet_bootstrap.cc:492] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0: Bootstrap starting.
I20260928 02:36:54.664319 14737 tablet_bootstrap.cc:654] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0: Neither blocks nor log segments found. Creating new log.
I20260928 02:36:54.664855 14737 tablet_bootstrap.cc:492] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0: No bootstrap required, opened a new log
I20260928 02:36:54.664906 14737 ts_tablet_manager.cc:1397] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 02:36:54.665243 14737 raft_consensus.cc:348] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } } peers { permanent_uuid: "fc893f328a02415b90838e91522ca56b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35121 } } peers { permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 } }
I20260928 02:36:54.665341 14737 raft_consensus.cc:374] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:36:54.665390 14737 raft_consensus.cc:729] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 589eafa5db924d669a9261c74dce4cb0, State: Initialized, Role: FOLLOWER
I20260928 02:36:54.665432 14737 consensus_queue.cc:260] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } } peers { permanent_uuid: "fc893f328a02415b90838e91522ca56b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35121 } } peers { permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 } }
I20260928 02:36:54.665522 14737 ts_tablet_manager.cc:1428] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 02:36:54.666291 15121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:54.662406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35121 (local address 127.0.0.1:51258)
0928 02:36:54.662552 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:54.662561 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:54.662599 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:54.662936 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:54.662937 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:54.662939 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:36:54.662978 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:54.662979 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.664174 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.664175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:54.665131 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:54.665135 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.665978 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.665979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:54.666101 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:54.666122 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 02:36:54.666176 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 02:36:54.666215 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
I20260928 02:36:54.666419 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:54.662315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36069 (local address 127.0.0.1:39932)
0928 02:36:54.662348 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:54.662358 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:54.662389 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:54.662896 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:54.662897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:54.662898 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:36:54.662938 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:54.662940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.664319 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.664320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:54.665332 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:54.665335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.666198 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.666198 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:54.666265 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:54.666270 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:36:54.666313 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 02:36:54.666351 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 02:36:54.666458 15122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:54.662424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39932 (local address 127.0.98.2:36069)
0928 02:36:54.662704 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:54.662710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:54.662732 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:36:54.662771 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:54.662773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:54.662789 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:54.663009 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:36:54.663079 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.664229 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.665426 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.666130 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.666196 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:54.666250 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:54.666341 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 02:36:54.666382 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":81,"threads_started":1}
I20260928 02:36:54.666544 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:54.662597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51258 (local address 127.0.98.3:35121)
0928 02:36:54.662774 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:54.662777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:54.662785 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:36:54.662818 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:54.662820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:54.662833 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:54.662943 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:36:54.663045 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.664073 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.665219 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.665911 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.665976 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:54.666033 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:54.666438 (+ 405us) server_negotiation.cc:299] Negotiation successful
0928 02:36:54.666476 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":86,"threads_started":1}
I20260928 02:36:54.667069 15049 tablet_service.cc:1467] Processing CreateTablet for tablet ebcbe086af124b8180ebc564f31ff7c9 (DEFAULT_TABLE table=test-table [id=cd50d5284def47aab77d3606c18ea215]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:36:54.667078 14917 tablet_service.cc:1467] Processing CreateTablet for tablet ebcbe086af124b8180ebc564f31ff7c9 (DEFAULT_TABLE table=test-table [id=cd50d5284def47aab77d3606c18ea215]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:36:54.667339 15049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebcbe086af124b8180ebc564f31ff7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:36:54.667349 14917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebcbe086af124b8180ebc564f31ff7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:36:54.669911 15125 tablet_bootstrap.cc:492] T ebcbe086af124b8180ebc564f31ff7c9 P ffe57bc5a65c4841aac9b43bf3205766: Bootstrap starting.
I20260928 02:36:54.669938 15126 tablet_bootstrap.cc:492] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b: Bootstrap starting.
I20260928 02:36:54.670478 15126 tablet_bootstrap.cc:654] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b: Neither blocks nor log segments found. Creating new log.
I20260928 02:36:54.670487 15125 tablet_bootstrap.cc:654] T ebcbe086af124b8180ebc564f31ff7c9 P ffe57bc5a65c4841aac9b43bf3205766: Neither blocks nor log segments found. Creating new log.
I20260928 02:36:54.670972 15126 log.cc:824] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:36:54.671000 15125 log.cc:824] T ebcbe086af124b8180ebc564f31ff7c9 P ffe57bc5a65c4841aac9b43bf3205766: Log is configured to *not* fsync() on all Append() calls
I20260928 02:36:54.671593 15126 tablet_bootstrap.cc:492] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b: No bootstrap required, opened a new log
I20260928 02:36:54.671623 15125 tablet_bootstrap.cc:492] T ebcbe086af124b8180ebc564f31ff7c9 P ffe57bc5a65c4841aac9b43bf3205766: No bootstrap required, opened a new log
I20260928 02:36:54.671644 15126 ts_tablet_manager.cc:1397] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:36:54.671677 15125 ts_tablet_manager.cc:1397] T ebcbe086af124b8180ebc564f31ff7c9 P ffe57bc5a65c4841aac9b43bf3205766: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:36:54.672899 15126 raft_consensus.cc:348] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } } peers { permanent_uuid: "fc893f328a02415b90838e91522ca56b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35121 } } peers { permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 } }
I20260928 02:36:54.672927 15125 raft_consensus.cc:348] T ebcbe086af124b8180ebc564f31ff7c9 P ffe57bc5a65c4841aac9b43bf3205766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } } peers { permanent_uuid: "fc893f328a02415b90838e91522ca56b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35121 } } peers { permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 } }
I20260928 02:36:54.673033 15126 raft_consensus.cc:374] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:36:54.673043 15125 raft_consensus.cc:374] T ebcbe086af124b8180ebc564f31ff7c9 P ffe57bc5a65c4841aac9b43bf3205766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:36:54.673054 15126 raft_consensus.cc:729] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc893f328a02415b90838e91522ca56b, State: Initialized, Role: FOLLOWER
I20260928 02:36:54.673070 15125 raft_consensus.cc:729] T ebcbe086af124b8180ebc564f31ff7c9 P ffe57bc5a65c4841aac9b43bf3205766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffe57bc5a65c4841aac9b43bf3205766, State: Initialized, Role: FOLLOWER
I20260928 02:36:54.673167 15126 consensus_queue.cc:260] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } } peers { permanent_uuid: "fc893f328a02415b90838e91522ca56b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35121 } } peers { permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 } }
I20260928 02:36:54.673177 15125 consensus_queue.cc:260] T ebcbe086af124b8180ebc564f31ff7c9 P ffe57bc5a65c4841aac9b43bf3205766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } } peers { permanent_uuid: "fc893f328a02415b90838e91522ca56b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35121 } } peers { permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 } }
I20260928 02:36:54.673458 15126 ts_tablet_manager.cc:1428] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:36:54.673458 15125 ts_tablet_manager.cc:1428] T ebcbe086af124b8180ebc564f31ff7c9 P ffe57bc5a65c4841aac9b43bf3205766: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 02:36:54.673485 14983 heartbeater.cc:503] Master 127.0.98.62:33383 was elected leader, sending a full tablet report...
I20260928 02:36:54.673614 15115 heartbeater.cc:503] Master 127.0.98.62:33383 was elected leader, sending a full tablet report...
W20260928 02:36:54.765765 14984 tablet.cc:2369] T ebcbe086af124b8180ebc564f31ff7c9 P ffe57bc5a65c4841aac9b43bf3205766: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:36:54.896065 15116 tablet.cc:2369] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:36:54.900486 14846 tablet.cc:2369] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:36:54.926829 15130 raft_consensus.cc:482] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:36:54.926952 15130 raft_consensus.cc:504] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } } peers { permanent_uuid: "fc893f328a02415b90838e91522ca56b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35121 } } peers { permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 } }
I20260928 02:36:54.927258 15130 leader_election.cc:288] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffe57bc5a65c4841aac9b43bf3205766 (127.0.98.2:36069), 589eafa5db924d669a9261c74dce4cb0 (127.0.98.1:37061)
I20260928 02:36:54.930938 15122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:54.927522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36557 (local address 127.0.98.2:36069)
0928 02:36:54.927602 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:54.927604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:54.927634 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:36:54.927681 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:54.927682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:54.927693 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:54.927743 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:36:54.927919 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.928648 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.930080 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.930618 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.930688 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:54.930698 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:54.930825 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 02:36:54.930858 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:36:54.930939 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:54.927443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36069 (local address 127.0.98.3:36557)
0928 02:36:54.927587 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:54.927596 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:54.927640 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:54.927753 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:54.927754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:54.927755 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:36:54.927850 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:54.927851 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.928715 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.928716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:54.929996 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:54.929999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.930686 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.930687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:54.930755 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:54.930760 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:36:54.930814 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 02:36:54.930853 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":24,"thread_start_us":88,"threads_started":1}
I20260928 02:36:54.931008 15117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:54.927428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37061 (local address 127.0.98.3:59443)
0928 02:36:54.927491 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:54.927503 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:54.927544 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:54.927753 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:54.927754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:54.927755 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:36:54.927849 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:54.927851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.928716 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.928716 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:54.929996 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:54.929999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.930731 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.930732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:54.930824 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:54.930827 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:36:54.930871 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 02:36:54.930903 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":84}
I20260928 02:36:54.930984 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:54.927497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59443 (local address 127.0.98.1:37061)
0928 02:36:54.927590 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:54.927592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:54.927603 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:36:54.927632 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:54.927633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:54.927661 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:54.927745 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:36:54.927914 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.928657 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.930070 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.930666 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.930721 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:54.930784 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:54.930881 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 02:36:54.930923 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":39,"threads_started":1}
I20260928 02:36:54.931329 14937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcbe086af124b8180ebc564f31ff7c9" candidate_uuid: "fc893f328a02415b90838e91522ca56b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe57bc5a65c4841aac9b43bf3205766" is_pre_election: true
I20260928 02:36:54.931352 14799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcbe086af124b8180ebc564f31ff7c9" candidate_uuid: "fc893f328a02415b90838e91522ca56b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "589eafa5db924d669a9261c74dce4cb0" is_pre_election: true
I20260928 02:36:54.931470 14937 raft_consensus.cc:2454] T ebcbe086af124b8180ebc564f31ff7c9 P ffe57bc5a65c4841aac9b43bf3205766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc893f328a02415b90838e91522ca56b in term 0.
I20260928 02:36:54.931474 14799 raft_consensus.cc:2454] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc893f328a02415b90838e91522ca56b in term 0.
I20260928 02:36:54.931663 15002 leader_election.cc:302] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 589eafa5db924d669a9261c74dce4cb0, fc893f328a02415b90838e91522ca56b; no voters:
I20260928 02:36:54.931787 15130 raft_consensus.cc:2781] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:36:54.931859 15130 raft_consensus.cc:482] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:36:54.931880 15130 raft_consensus.cc:3037] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:36:54.932549 15130 raft_consensus.cc:504] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } } peers { permanent_uuid: "fc893f328a02415b90838e91522ca56b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35121 } } peers { permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 } }
I20260928 02:36:54.932662 15130 leader_election.cc:288] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b [CANDIDATE]: Term 1 election: Requested vote from peers ffe57bc5a65c4841aac9b43bf3205766 (127.0.98.2:36069), 589eafa5db924d669a9261c74dce4cb0 (127.0.98.1:37061)
I20260928 02:36:54.932857 14937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcbe086af124b8180ebc564f31ff7c9" candidate_uuid: "fc893f328a02415b90838e91522ca56b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffe57bc5a65c4841aac9b43bf3205766"
I20260928 02:36:54.932796 14799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcbe086af124b8180ebc564f31ff7c9" candidate_uuid: "fc893f328a02415b90838e91522ca56b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "589eafa5db924d669a9261c74dce4cb0"
I20260928 02:36:54.932935 14799 raft_consensus.cc:3037] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:36:54.932936 14937 raft_consensus.cc:3037] T ebcbe086af124b8180ebc564f31ff7c9 P ffe57bc5a65c4841aac9b43bf3205766 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:36:54.933759 14799 raft_consensus.cc:2454] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc893f328a02415b90838e91522ca56b in term 1.
I20260928 02:36:54.933759 14937 raft_consensus.cc:2454] T ebcbe086af124b8180ebc564f31ff7c9 P ffe57bc5a65c4841aac9b43bf3205766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc893f328a02415b90838e91522ca56b in term 1.
I20260928 02:36:54.933923 15003 leader_election.cc:302] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fc893f328a02415b90838e91522ca56b, ffe57bc5a65c4841aac9b43bf3205766; no voters:
I20260928 02:36:54.934029 15130 raft_consensus.cc:2781] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:36:54.934155 15130 raft_consensus.cc:686] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b [term 1 LEADER]: Becoming Leader. State: Replica: fc893f328a02415b90838e91522ca56b, State: Running, Role: LEADER
I20260928 02:36:54.934240 15130 consensus_queue.cc:237] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } } peers { permanent_uuid: "fc893f328a02415b90838e91522ca56b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35121 } } peers { permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 } }
I20260928 02:36:54.934966 14657 catalog_manager.cc:5495] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b reported cstate change: term changed from 0 to 1, leader changed from <none> to fc893f328a02415b90838e91522ca56b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "fc893f328a02415b90838e91522ca56b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc893f328a02415b90838e91522ca56b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 } health_report { overall_health: UNKNOWN } } }
I20260928 02:36:54.948117 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:54.945065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51266 (local address 127.0.98.3:35121)
0928 02:36:54.945103 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:54.945105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:54.945112 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:36:54.945159 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:54.945160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:54.945179 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:54.945240 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:36:54.945380 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.946085 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.947095 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:54.947386 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:54.947465 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:54.947666 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:36:54.947799 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:36:54.947890 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:54.948011 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 02:36:54.948038 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:36:54.949992 14937 raft_consensus.cc:1261] T ebcbe086af124b8180ebc564f31ff7c9 P ffe57bc5a65c4841aac9b43bf3205766 [term 1 FOLLOWER]: Refusing update from remote peer fc893f328a02415b90838e91522ca56b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:36:54.950006 14799 raft_consensus.cc:1261] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0 [term 1 FOLLOWER]: Refusing update from remote peer fc893f328a02415b90838e91522ca56b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:36:54.950332 15133 consensus_queue.cc:1035] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b [LEADER]: Connected to new peer: Peer: permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:36:54.950454 15130 consensus_queue.cc:1035] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:36:54.952766 15138 mvcc.cc:205] Tried to move back new op lower bound from 7334146109232783360 to 7334146109171228672. Current Snapshot: MvccSnapshot[applied={T|T < 7334146109232783360}]
I20260928 02:36:54.953176 15137 mvcc.cc:205] Tried to move back new op lower bound from 7334146109232783360 to 7334146109171228672. Current Snapshot: MvccSnapshot[applied={T|T < 7334146109232783360}]
I20260928 02:36:54.954504 14657 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:39446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:36:54.954576 14657 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:39446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:36:54.955210 14657 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13: Sending DeleteTablet for 3 replicas of tablet ebcbe086af124b8180ebc564f31ff7c9
I20260928 02:36:54.955463 14777 tablet_service.cc:1514] Processing DeleteTablet for tablet ebcbe086af124b8180ebc564f31ff7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:36:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35696
I20260928 02:36:54.955582 14917 tablet_service.cc:1514] Processing DeleteTablet for tablet ebcbe086af124b8180ebc564f31ff7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:36:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39932
I20260928 02:36:54.955667 15145 tablet_replica.cc:330] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0: stopping tablet replica
I20260928 02:36:54.955698 15049 tablet_service.cc:1514] Processing DeleteTablet for tablet ebcbe086af124b8180ebc564f31ff7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:36:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51258
I20260928 02:36:54.955722 15145 raft_consensus.cc:2229] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:36:54.955763 15145 raft_consensus.cc:2258] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:36:54.955772 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 14695
I20260928 02:36:54.955987 15147 tablet_replica.cc:330] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b: stopping tablet replica
I20260928 02:36:54.956096 15147 raft_consensus.cc:2229] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:36:54.956218 15147 raft_consensus.cc:2258] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:36:54.956598 15147 ts_tablet_manager.cc:1905] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:36:54.958220 15147 ts_tablet_manager.cc:1918] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:36:54.958328 15147 log.cc:1196] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/wal/wals/ebcbe086af124b8180ebc564f31ff7c9
I20260928 02:36:54.958602 15147 ts_tablet_manager.cc:1939] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b: Deleting consensus metadata
I20260928 02:36:54.958968 14642 catalog_manager.cc:4841] TS fc893f328a02415b90838e91522ca56b (127.0.98.3:35121): tablet ebcbe086af124b8180ebc564f31ff7c9 (table test-table [id=cd50d5284def47aab77d3606c18ea215]) successfully deleted
I20260928 02:36:54.961383 15146 tablet_replica.cc:330] T ebcbe086af124b8180ebc564f31ff7c9 P ffe57bc5a65c4841aac9b43bf3205766: stopping tablet replica
I20260928 02:36:54.961483 15146 raft_consensus.cc:2229] T ebcbe086af124b8180ebc564f31ff7c9 P ffe57bc5a65c4841aac9b43bf3205766 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:36:54.961582 15146 raft_consensus.cc:2258] T ebcbe086af124b8180ebc564f31ff7c9 P ffe57bc5a65c4841aac9b43bf3205766 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 02:36:54.961680 14641 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37061 (error 108)
I20260928 02:36:54.962073 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 14849
I20260928 02:36:54.962309 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:54.962203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37061 (local address 127.0.0.1:35702)
0928 02:36:54.962247 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:54.962280 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 02:36:54.962400 14641 catalog_manager.cc:4584] TS 589eafa5db924d669a9261c74dce4cb0 (127.0.98.1:37061): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ebcbe086af124b8180ebc564f31ff7c9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37061: connect: Connection refused (error 111)
I20260928 02:36:54.966977 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 14986
I20260928 02:36:54.967046 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:54.966949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36069 (local address 127.0.0.1:39940)
0928 02:36:54.966990 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:54.967017 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 02:36:54.970999 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 14624
I20260928 02:36:54.975124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33383
--webserver_interface=127.0.98.62
--webserver_port=43781
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:36:55.053041 15149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:36:55.053233 15149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:36:55.053282 15149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:36:55.054431 15149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:36:55.054472 15149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:36:55.054490 15149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:36:55.054504 15149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:36:55.054520 15149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:36:55.056006 15149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:36:55.056241 15149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:36:55.056438 15149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:36:55.060871 15155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:55.060937 15149 server_base.cc:1034] running on GCE node
W20260928 02:36:55.060988 15157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:55.061054 15154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:55.061479 15149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:36:55.061734 15149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:36:55.062870 15149 hybrid_clock.cc:648] HybridClock initialized: now 1790563015062855 us; error 30 us; skew 500 ppm
Sep 28 02:36:55 dist-test-slave-fz47 krb5kdc[14080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563015, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:36:55.065056 15149 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:36:55.065500 15149 webserver.cc:460] Webserver started at http://127.0.98.62:43781/ using document root <none> and password file <none>
I20260928 02:36:55.065702 15149 fs_manager.cc:359] Metadata directory not provided
I20260928 02:36:55.065744 15149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:36:55.067041 15149 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:36:55.067598 15164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:55.067756 15149 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:36:55.067833 15149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "34384d463a834c0ea4abe376446aab13"
format_stamp: "Formatted at 2026-09-28 02:36:53 on dist-test-slave-fz47"
server_key: "a502da60ccceaebb557b50273b4101b8"
server_key_iv: "b9cc61d58beb3be4c243215166552638"
server_key_version: "encryptionkey@0"
I20260928 02:36:55.068137 15149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:36:55.095069 15149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:36:55.096886 15149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:36:55.097019 15149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:36:55.100935 15149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33383
I20260928 02:36:55.100978 15216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33383 every 8 connection(s)
I20260928 02:36:55.101329 15149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:36:55.103252 15217 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:36:55.103984 15217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13: Bootstrap starting.
I20260928 02:36:55.109607 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 15149
I20260928 02:36:55.109911 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37061
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33383
--webserver_port=44669
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:36:55.112423 15217 log.cc:824] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13: Log is configured to *not* fsync() on all Append() calls
I20260928 02:36:55.114723 15217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:36:55.114997 15217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13: Bootstrap complete.
I20260928 02:36:55.117023 15217 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34384d463a834c0ea4abe376446aab13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33383 } }
I20260928 02:36:55.117235 15217 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34384d463a834c0ea4abe376446aab13, State: Initialized, Role: FOLLOWER
I20260928 02:36:55.117393 15217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34384d463a834c0ea4abe376446aab13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33383 } }
I20260928 02:36:55.117468 15217 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:36:55.117499 15217 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:36:55.117545 15217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [term 2 FOLLOWER]: Advancing to term 3
I20260928 02:36:55.118472 15217 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34384d463a834c0ea4abe376446aab13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33383 } }
I20260928 02:36:55.118556 15217 leader_election.cc:302] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34384d463a834c0ea4abe376446aab13; no voters:
I20260928 02:36:55.118698 15217 leader_election.cc:288] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 02:36:55.118750 15221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 02:36:55.118866 15221 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [term 3 LEADER]: Becoming Leader. State: Replica: 34384d463a834c0ea4abe376446aab13, State: Running, Role: LEADER
I20260928 02:36:55.118957 15217 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:36:55.118947 15221 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34384d463a834c0ea4abe376446aab13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33383 } }
I20260928 02:36:55.119215 15221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34384d463a834c0ea4abe376446aab13. Latest consensus state: current_term: 3 leader_uuid: "34384d463a834c0ea4abe376446aab13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34384d463a834c0ea4abe376446aab13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33383 } } }
I20260928 02:36:55.119297 15221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [sys.catalog]: This master's current role is: LEADER
I20260928 02:36:55.119218 15223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "34384d463a834c0ea4abe376446aab13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34384d463a834c0ea4abe376446aab13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33383 } } }
I20260928 02:36:55.119438 15223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34384d463a834c0ea4abe376446aab13 [sys.catalog]: This master's current role is: LEADER
I20260928 02:36:55.119586 15226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:36:55.120368 15226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:36:55.120464 15226 catalog_manager.cc:1259] Loaded cluster ID: 7ab64556213d428eaaf93afd40278b18
I20260928 02:36:55.120496 15226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 02:36:55.120913 15226 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fcc28cd79ad google::LogMessage::Fail() at ??:0
@ 0x7fcc28cda88a google::LogMessage::SendToLog() at ??:0
@ 0x7fcc28cd74de google::LogMessage::Flush() at ??:0
@ 0x7fcc28cd8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fcc30bf80f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fcc30c018b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fcc2aba50d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fcc2ab9a954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fcc2b9fe6db start_thread at ??:0
@ 0x7fcc277d561f clone at ??:0
W20260928 02:36:55.224073 15219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:36:55.224279 15219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:36:55.224303 15219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:36:55.225878 15219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:36:55.225934 15219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:36:55.225973 15219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:36:55.228070 15219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37061
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44669
--tserver_master_addrs=127.0.98.62:33383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:36:55.228329 15219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:36:55.228606 15219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:36:55.233670 15253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:55.233701 15254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:55.233896 15256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:55.233790 15219 server_base.cc:1034] running on GCE node
I20260928 02:36:55.234088 15219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:36:55.234354 15219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:36:55.235519 15219 hybrid_clock.cc:648] HybridClock initialized: now 1790563015235502 us; error 32 us; skew 500 ppm
Sep 28 02:36:55 dist-test-slave-fz47 krb5kdc[14080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563015, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:36:55.237962 15219 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 02:36:55.238463 15219 webserver.cc:460] Webserver started at http://127.0.98.1:44669/ using document root <none> and password file <none>
I20260928 02:36:55.238633 15219 fs_manager.cc:359] Metadata directory not provided
I20260928 02:36:55.238672 15219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:36:55.240074 15219 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:36:55.240801 15263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:55.240958 15219 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:36:55.241046 15219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "589eafa5db924d669a9261c74dce4cb0"
format_stamp: "Formatted at 2026-09-28 02:36:53 on dist-test-slave-fz47"
server_key: "dd33b1baca925e99708089302a005fae"
server_key_iv: "39c70270a11f9d4ef3fdac9f24aa809c"
server_key_version: "encryptionkey@0"
I20260928 02:36:55.241537 15219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:36:55.252768 15219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:36:55.254983 15219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:36:55.255118 15219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:36:55.255330 15219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:36:55.255751 15270 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 02:36:55.257732 15219 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:36:55.257779 15219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 02:36:55.257812 15219 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:36:55.258433 15219 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:36:55.258466 15219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 02:36:55.258558 15270 tablet_bootstrap.cc:492] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0: Bootstrap starting.
I20260928 02:36:55.264845 15219 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37061
I20260928 02:36:55.265211 15219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
I20260928 02:36:55.265615 15377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37061 every 8 connection(s)
I20260928 02:36:55.269290 15270 log.cc:824] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0: Log is configured to *not* fsync() on all Append() calls
I20260928 02:36:55.271986 15270 tablet_bootstrap.cc:492] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:36:55.272284 15270 tablet_bootstrap.cc:492] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0: Bootstrap complete.
I20260928 02:36:55.273391 15270 ts_tablet_manager.cc:1397] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.010s
I20260928 02:36:55.274813 15270 raft_consensus.cc:348] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } } peers { permanent_uuid: "fc893f328a02415b90838e91522ca56b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35121 } } peers { permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 } }
I20260928 02:36:55.275178 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 15219
I20260928 02:36:55.275190 15270 raft_consensus.cc:729] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 589eafa5db924d669a9261c74dce4cb0, State: Initialized, Role: FOLLOWER
I20260928 02:36:55.275344 15270 consensus_queue.cc:260] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } } peers { permanent_uuid: "fc893f328a02415b90838e91522ca56b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35121 } } peers { permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 } }
I20260928 02:36:55.275411 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36069
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33383
--webserver_port=36877
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:36:55.275600 15270 ts_tablet_manager.cc:1428] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20260928 02:36:55.337312 15380 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:36:55.265893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.1:46657)
0928 02:36:55.266077 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:55.266111 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:55.266539 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:55.337074 (+ 70535us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":107,"threads_started":1}
W20260928 02:36:55.337566 15378 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33383 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33383: Client connection negotiation failed: client connection to 127.0.98.62:33383: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:36:55.337908 15380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:55.337804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.1:53275)
0928 02:36:55.337850 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:55.337887 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 02:36:55.338231 15380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:55.338155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.1:33239)
0928 02:36:55.338196 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:55.338217 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 02:36:55.338342 15378 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:36:55.354007 15383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:36:55.354189 15383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:36:55.354211 15383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:36:55.355352 15383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:36:55.355391 15383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:36:55.355417 15383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:36:55.356808 15383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36069
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36877
--tserver_master_addrs=127.0.98.62:33383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:36:55.356994 15383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:36:55.357168 15383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:36:55.361809 15388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:55.361928 15383 server_base.cc:1034] running on GCE node
W20260928 02:36:55.361905 15389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:55.361820 15391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:55.362190 15383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:36:55.362383 15383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:36:55.363515 15383 hybrid_clock.cc:648] HybridClock initialized: now 1790563015363503 us; error 24 us; skew 500 ppm
Sep 28 02:36:55 dist-test-slave-fz47 krb5kdc[14080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563015, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:36:55.365748 15383 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 02:36:55.366161 15383 webserver.cc:460] Webserver started at http://127.0.98.2:36877/ using document root <none> and password file <none>
I20260928 02:36:55.366360 15383 fs_manager.cc:359] Metadata directory not provided
I20260928 02:36:55.366411 15383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:36:55.367588 15383 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:36:55.368135 15398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:55.368270 15383 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:55.368345 15383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "ffe57bc5a65c4841aac9b43bf3205766"
format_stamp: "Formatted at 2026-09-28 02:36:53 on dist-test-slave-fz47"
server_key: "237f17fa9fef4f79568b239884002003"
server_key_iv: "b3de135f1dc25a5092746ea5ef887817"
server_key_version: "encryptionkey@0"
I20260928 02:36:55.368649 15383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:36:55.377882 15383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:36:55.380251 15383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:36:55.380379 15383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:36:55.380585 15383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:36:55.381055 15405 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 02:36:55.382462 15383 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:36:55.382499 15383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 02:36:55.382524 15383 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:36:55.383162 15383 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:36:55.383198 15383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 02:36:55.383273 15405 tablet_bootstrap.cc:492] T ebcbe086af124b8180ebc564f31ff7c9 P ffe57bc5a65c4841aac9b43bf3205766: Bootstrap starting.
I20260928 02:36:55.389745 15383 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36069
I20260928 02:36:55.389892 15512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36069 every 8 connection(s)
I20260928 02:36:55.390138 15383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
I20260928 02:36:55.391162 15515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:55.390784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.2:44967)
0928 02:36:55.390957 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:55.391037 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":66,"threads_started":1}
W20260928 02:36:55.391357 15513 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33383 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33383: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
I20260928 02:36:55.391623 15515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:55.391542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.2:33789)
0928 02:36:55.391582 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:55.391607 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 02:36:55.391901 15515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:55.391816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.2:60269)
0928 02:36:55.391839 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:55.391855 (+ 16us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260928 02:36:55.391860 15405 log.cc:824] T ebcbe086af124b8180ebc564f31ff7c9 P ffe57bc5a65c4841aac9b43bf3205766: Log is configured to *not* fsync() on all Append() calls
W20260928 02:36:55.391993 15513 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:36:55.393865 15405 tablet_bootstrap.cc:492] T ebcbe086af124b8180ebc564f31ff7c9 P ffe57bc5a65c4841aac9b43bf3205766: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:36:55.394093 15405 tablet_bootstrap.cc:492] T ebcbe086af124b8180ebc564f31ff7c9 P ffe57bc5a65c4841aac9b43bf3205766: Bootstrap complete.
I20260928 02:36:55.394845 15405 ts_tablet_manager.cc:1397] T ebcbe086af124b8180ebc564f31ff7c9 P ffe57bc5a65c4841aac9b43bf3205766: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260928 02:36:55.395969 15405 raft_consensus.cc:348] T ebcbe086af124b8180ebc564f31ff7c9 P ffe57bc5a65c4841aac9b43bf3205766 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } } peers { permanent_uuid: "fc893f328a02415b90838e91522ca56b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35121 } } peers { permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 } }
I20260928 02:36:55.396205 15405 raft_consensus.cc:729] T ebcbe086af124b8180ebc564f31ff7c9 P ffe57bc5a65c4841aac9b43bf3205766 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffe57bc5a65c4841aac9b43bf3205766, State: Initialized, Role: FOLLOWER
I20260928 02:36:55.396332 15405 consensus_queue.cc:260] T ebcbe086af124b8180ebc564f31ff7c9 P ffe57bc5a65c4841aac9b43bf3205766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } } peers { permanent_uuid: "fc893f328a02415b90838e91522ca56b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35121 } } peers { permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 } }
I20260928 02:36:55.396543 15405 ts_tablet_manager.cc:1428] T ebcbe086af124b8180ebc564f31ff7c9 P ffe57bc5a65c4841aac9b43bf3205766: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:36:55.396760 15515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:55.396680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.2:33007)
0928 02:36:55.396717 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:55.396741 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 02:36:55.396961 15515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:55.396913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.2:45781)
0928 02:36:55.396934 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:55.396950 (+ 16us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
I20260928 02:36:55.400098 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 15383
I20260928 02:36:55.400317 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35121
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33383
--webserver_port=39447
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:36:55.478160 15518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:36:55.478351 15518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:36:55.478372 15518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:36:55.479477 15518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:36:55.479516 15518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:36:55.479544 15518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:36:55.480958 15518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35121
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39447
--tserver_master_addrs=127.0.98.62:33383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:36:55.481168 15518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:36:55.481391 15518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:36:55.485867 15524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:36:55.485882 15526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:55.486078 15518 server_base.cc:1034] running on GCE node
W20260928 02:36:55.485882 15523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:36:55.486370 15518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:36:55.486572 15518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:36:55.487727 15518 hybrid_clock.cc:648] HybridClock initialized: now 1790563015487710 us; error 30 us; skew 500 ppm
Sep 28 02:36:55 dist-test-slave-fz47 krb5kdc[14080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563015, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:36:55.490207 15518 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 02:36:55.490639 15518 webserver.cc:460] Webserver started at http://127.0.98.3:39447/ using document root <none> and password file <none>
I20260928 02:36:55.490844 15518 fs_manager.cc:359] Metadata directory not provided
I20260928 02:36:55.490890 15518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:36:55.492122 15518 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:36:55.492633 15533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:55.492805 15518 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:55.492884 15518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "fc893f328a02415b90838e91522ca56b"
format_stamp: "Formatted at 2026-09-28 02:36:53 on dist-test-slave-fz47"
server_key: "7db6c44f0bd4d0ded2b354e4774713f6"
server_key_iv: "7b68eae3cea99d4890d3a6bb6a033c53"
server_key_version: "encryptionkey@0"
I20260928 02:36:55.493204 15518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:36:55.509425 15518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:36:55.511190 15518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:36:55.511318 15518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:36:55.511528 15518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:36:55.511813 15518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:36:55.511850 15518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:55.511879 15518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:36:55.511904 15518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:36:55.517225 15518 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35121
I20260928 02:36:55.517307 15646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35121 every 8 connection(s)
I20260928 02:36:55.517601 15518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
I20260928 02:36:55.518512 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:55.518168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.3:54159)
0928 02:36:55.518314 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:55.518379 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
W20260928 02:36:55.518885 15647 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33383 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33383: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
I20260928 02:36:55.519194 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:55.519047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.3:53773)
0928 02:36:55.519117 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:55.519151 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 02:36:55.519462 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:55.519407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.3:43617)
0928 02:36:55.519428 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:55.519448 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260928 02:36:55.519541 15647 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:36:55.524438 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 15518
I20260928 02:36:56.338891 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:56.338676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.1:41595)
0928 02:36:56.338790 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:56.338830 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":62,"threads_started":1}
I20260928 02:36:56.397558 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:56.397364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.2:57645)
0928 02:36:56.397473 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:56.397508 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":54,"threads_started":1}
I20260928 02:36:56.439815 15653 raft_consensus.cc:482] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:36:56.439945 15653 raft_consensus.cc:504] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } } peers { permanent_uuid: "fc893f328a02415b90838e91522ca56b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35121 } } peers { permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 } }
I20260928 02:36:56.440284 15653 leader_election.cc:288] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ffe57bc5a65c4841aac9b43bf3205766 (127.0.98.2:36069), fc893f328a02415b90838e91522ca56b (127.0.98.3:35121)
Sep 28 02:36:56 dist-test-slave-fz47 krb5kdc[14080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563015, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 28 02:36:56 dist-test-slave-fz47 krb5kdc[14080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563015, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260928 02:36:56.454734 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:56.440768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49691 (local address 127.0.98.3:35121)
0928 02:36:56.440884 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:56.440894 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:56.440912 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:36:56.441349 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:56.441352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:56.441405 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:56.441527 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:56.441858 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:56.442799 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:56.444616 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:56.445289 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:56.445378 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:56.451429 (+ 6051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:56.451434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:56.451434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:56.451447 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:56.453227 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:56.453596 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:56.453599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:56.453600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:56.453693 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:56.453870 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:56.453871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:56.453875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:56.454158 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:56.454219 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:56.454498 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 02:36:56.454616 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":52,"threads_started":1}
I20260928 02:36:56.454766 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:56.440529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35121 (local address 127.0.98.1:49691)
0928 02:36:56.440635 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:56.440645 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:56.440688 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:56.441515 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:56.441516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:56.441695 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:36:56.441743 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:56.441746 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:56.442885 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:56.442886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:56.443968 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:56.443970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:56.445354 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:56.445355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:56.445401 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:56.449349 (+ 3948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:36:56.449359 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:36:56.451332 (+ 1973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:36:56.453337 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:56.453339 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:56.453339 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:56.453526 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:56.453758 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:56.453759 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:56.453760 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:56.453813 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:56.454230 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:36:56.454237 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:36:56.454353 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 02:36:56.454434 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 02:36:56.454639 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":27,"thread_start_us":61,"threads_started":1}
I20260928 02:36:56.454838 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:56.440431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36069 (local address 127.0.98.1:44837)
0928 02:36:56.440478 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:56.440491 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:56.440562 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:56.441036 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:56.441039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:56.441391 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:36:56.441577 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:56.441583 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:56.442763 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:56.442764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:56.443968 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:56.443970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:56.444529 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:56.444530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:56.444617 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:56.449349 (+ 4732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:36:56.449359 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:36:56.450975 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:36:56.453136 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:56.453143 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:56.453144 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:56.453449 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:56.453693 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:56.453694 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:56.453694 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:56.453799 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:56.454233 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:36:56.454241 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:36:56.454354 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 02:36:56.454433 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 02:36:56.454697 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":154}
I20260928 02:36:56.454762 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:56.440553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44837 (local address 127.0.98.2:36069)
0928 02:36:56.440675 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:56.440687 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:56.440713 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:36:56.440840 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:56.440841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:56.440881 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:56.440989 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:56.441673 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:56.442672 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:56.444050 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:56.444452 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:56.444539 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:56.451091 (+ 6552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:56.451096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:56.451097 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:56.451110 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:56.453048 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:56.453529 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:56.453533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:56.453533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:56.453629 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:56.453863 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:56.453864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:56.453864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:56.454172 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:56.454245 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:56.454506 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 02:36:56.454643 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":55,"threads_started":1}
W20260928 02:36:56.455485 15265 leader_election.cc:341] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fc893f328a02415b90838e91522ca56b (127.0.98.3:35121): Not found: Tablet not found: ebcbe086af124b8180ebc564f31ff7c9
I20260928 02:36:56.455718 15467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcbe086af124b8180ebc564f31ff7c9" candidate_uuid: "589eafa5db924d669a9261c74dce4cb0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ffe57bc5a65c4841aac9b43bf3205766" is_pre_election: true
I20260928 02:36:56.455847 15467 raft_consensus.cc:2454] T ebcbe086af124b8180ebc564f31ff7c9 P ffe57bc5a65c4841aac9b43bf3205766 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 589eafa5db924d669a9261c74dce4cb0 in term 1.
I20260928 02:36:56.456063 15265 leader_election.cc:302] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 589eafa5db924d669a9261c74dce4cb0, ffe57bc5a65c4841aac9b43bf3205766; no voters: fc893f328a02415b90838e91522ca56b
I20260928 02:36:56.456193 15653 raft_consensus.cc:2781] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 02:36:56.456260 15653 raft_consensus.cc:482] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:36:56.456277 15653 raft_consensus.cc:3037] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:36:56.457405 15653 raft_consensus.cc:504] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } } peers { permanent_uuid: "fc893f328a02415b90838e91522ca56b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35121 } } peers { permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 } }
I20260928 02:36:56.457527 15653 leader_election.cc:288] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0 [CANDIDATE]: Term 2 election: Requested vote from peers ffe57bc5a65c4841aac9b43bf3205766 (127.0.98.2:36069), fc893f328a02415b90838e91522ca56b (127.0.98.3:35121)
I20260928 02:36:56.457757 15467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcbe086af124b8180ebc564f31ff7c9" candidate_uuid: "589eafa5db924d669a9261c74dce4cb0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ffe57bc5a65c4841aac9b43bf3205766"
I20260928 02:36:56.457824 15467 raft_consensus.cc:3037] T ebcbe086af124b8180ebc564f31ff7c9 P ffe57bc5a65c4841aac9b43bf3205766 [term 1 FOLLOWER]: Advancing to term 2
W20260928 02:36:56.457866 15265 leader_election.cc:341] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer fc893f328a02415b90838e91522ca56b (127.0.98.3:35121): Not found: Tablet not found: ebcbe086af124b8180ebc564f31ff7c9
I20260928 02:36:56.458824 15467 raft_consensus.cc:2454] T ebcbe086af124b8180ebc564f31ff7c9 P ffe57bc5a65c4841aac9b43bf3205766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 589eafa5db924d669a9261c74dce4cb0 in term 2.
I20260928 02:36:56.458995 15265 leader_election.cc:302] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 589eafa5db924d669a9261c74dce4cb0, ffe57bc5a65c4841aac9b43bf3205766; no voters: fc893f328a02415b90838e91522ca56b
I20260928 02:36:56.459102 15653 raft_consensus.cc:2781] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:36:56.459220 15653 raft_consensus.cc:686] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0 [term 2 LEADER]: Becoming Leader. State: Replica: 589eafa5db924d669a9261c74dce4cb0, State: Running, Role: LEADER
I20260928 02:36:56.459311 15653 consensus_queue.cc:237] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } } peers { permanent_uuid: "fc893f328a02415b90838e91522ca56b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35121 } } peers { permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 } }
I20260928 02:36:56.459520 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:56.459406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.1:56709)
0928 02:36:56.459445 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:56.459494 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 02:36:56.520602 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:56.520125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.3:36071)
0928 02:36:56.520460 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:56.520546 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":114,"threads_started":1}
W20260928 02:36:56.902447 15265 consensus_peers.cc:487] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0 -> Peer fc893f328a02415b90838e91522ca56b (127.0.98.3:35121): Couldn't send request to peer fc893f328a02415b90838e91522ca56b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ebcbe086af124b8180ebc564f31ff7c9. This is attempt 1: this message will repeat every 5th retry.
I20260928 02:36:57.043496 15467 raft_consensus.cc:1261] T ebcbe086af124b8180ebc564f31ff7c9 P ffe57bc5a65c4841aac9b43bf3205766 [term 2 FOLLOWER]: Refusing update from remote peer 589eafa5db924d669a9261c74dce4cb0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260928 02:36:57.043843 15653 consensus_queue.cc:1035] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260928 02:36:57.044691 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:57.044545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.2:58589)
0928 02:36:57.044621 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:57.044656 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65,"thread_start_us":39,"threads_started":1}
I20260928 02:36:57.045511 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:57.045327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.1:34593)
0928 02:36:57.045450 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:57.045479 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":47,"threads_started":1}
I20260928 02:36:57.045567 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:57.045457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.2:58859)
0928 02:36:57.045496 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:57.045521 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 02:36:57.294624 15667 ts_tablet_manager.cc:927] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b: Initiating tablet copy from peer 589eafa5db924d669a9261c74dce4cb0 (127.0.98.1:37061)
I20260928 02:36:57.294832 15667 tablet_copy_client.cc:316] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.1:37061
Sep 28 02:36:57 dist-test-slave-fz47 krb5kdc[14080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563015, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 02:36:57.301939 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:57.295087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37061 (local address 127.0.98.3:50173)
0928 02:36:57.295246 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:57.295263 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:36:57.295326 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:36:57.295481 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:36:57.295483 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:36:57.295756 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:36:57.295824 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:57.295826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:57.296666 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:57.296667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:57.297302 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:36:57.297305 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:36:57.297745 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:36:57.297745 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:36:57.297812 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:57.298237 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:36:57.298242 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:36:57.299471 (+ 1229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:36:57.300954 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:57.300960 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:57.300961 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:57.301157 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:57.301342 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:36:57.301343 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:36:57.301343 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:36:57.301395 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:36:57.301685 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:36:57.301690 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:36:57.301732 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 02:36:57.301790 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 02:36:57.301870 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":82,"threads_started":1}
I20260928 02:36:57.301949 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:57.295180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50173 (local address 127.0.98.1:37061)
0928 02:36:57.295357 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 02:36:57.295362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:36:57.295372 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:36:57.295395 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:36:57.295397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:36:57.295408 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:36:57.295472 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:36:57.295892 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:57.296585 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:57.297368 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:36:57.297686 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:36:57.297738 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:36:57.299627 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:36:57.299637 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:36:57.299638 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:36:57.299647 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:36:57.300888 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:57.301216 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:57.301218 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:57.301218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:57.301293 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:36:57.301467 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:36:57.301468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:36:57.301468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:36:57.301632 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:36:57.301700 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 02:36:57.301821 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 02:36:57.301883 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":100,"threads_started":1}
I20260928 02:36:57.302348 15352 tablet_copy_service.cc:140] P 589eafa5db924d669a9261c74dce4cb0: Received BeginTabletCopySession request for tablet ebcbe086af124b8180ebc564f31ff7c9 from peer fc893f328a02415b90838e91522ca56b ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50173)
I20260928 02:36:57.302409 15352 tablet_copy_service.cc:161] P 589eafa5db924d669a9261c74dce4cb0: Beginning new tablet copy session on tablet ebcbe086af124b8180ebc564f31ff7c9 from peer fc893f328a02415b90838e91522ca56b at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50173: session id = fc893f328a02415b90838e91522ca56b-ebcbe086af124b8180ebc564f31ff7c9
I20260928 02:36:57.302917 15352 tablet_copy_source_session.cc:205] T ebcbe086af124b8180ebc564f31ff7c9 P 589eafa5db924d669a9261c74dce4cb0: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 02:36:57.303791 15667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebcbe086af124b8180ebc564f31ff7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:36:57.306207 15667 tablet_copy_client.cc:791] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b: tablet copy: Starting download of 0 data blocks...
I20260928 02:36:57.306336 15667 tablet_copy_client.cc:655] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b: tablet copy: Starting download of 1 WAL segments...
I20260928 02:36:57.307116 15667 tablet_copy_client.cc:531] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 02:36:57.307797 15667 tablet_bootstrap.cc:492] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b: Bootstrap starting.
I20260928 02:36:57.309027 15667 log.cc:824] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:36:57.311323 15667 tablet_bootstrap.cc:492] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:36:57.311502 15667 tablet_bootstrap.cc:492] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b: Bootstrap complete.
I20260928 02:36:57.311614 15667 ts_tablet_manager.cc:1397] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 02:36:57.312492 15667 raft_consensus.cc:348] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } } peers { permanent_uuid: "fc893f328a02415b90838e91522ca56b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35121 } } peers { permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 } }
I20260928 02:36:57.312635 15667 raft_consensus.cc:729] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc893f328a02415b90838e91522ca56b, State: Initialized, Role: FOLLOWER
I20260928 02:36:57.312741 15667 consensus_queue.cc:260] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ffe57bc5a65c4841aac9b43bf3205766" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36069 } } peers { permanent_uuid: "fc893f328a02415b90838e91522ca56b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35121 } } peers { permanent_uuid: "589eafa5db924d669a9261c74dce4cb0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37061 } }
I20260928 02:36:57.312948 15667 ts_tablet_manager.cc:1428] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20260928 02:36:57.313203 15352 tablet_copy_service.cc:342] P 589eafa5db924d669a9261c74dce4cb0: Request end of tablet copy session fc893f328a02415b90838e91522ca56b-ebcbe086af124b8180ebc564f31ff7c9 received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50173
I20260928 02:36:57.313233 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:57.313142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.3:51429)
0928 02:36:57.313182 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:57.313205 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 02:36:57.313288 15352 tablet_copy_service.cc:433] P 589eafa5db924d669a9261c74dce4cb0: ending tablet copy session fc893f328a02415b90838e91522ca56b-ebcbe086af124b8180ebc564f31ff7c9 on tablet ebcbe086af124b8180ebc564f31ff7c9 with peer fc893f328a02415b90838e91522ca56b
I20260928 02:36:57.314260 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:57.314188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.3:35989)
0928 02:36:57.314215 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:57.314240 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260928 02:36:57.785069 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:57.784846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.1:50783)
0928 02:36:57.784966 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:57.785024 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":54,"threads_started":1}
I20260928 02:36:57.785020 15601 raft_consensus.cc:1203] T ebcbe086af124b8180ebc564f31ff7c9 P fc893f328a02415b90838e91522ca56b [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260928 02:36:57.785490 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:57.785395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.3:59099)
0928 02:36:57.785435 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:57.785464 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 02:36:58.046100 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:58.045911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.2:48583)
0928 02:36:58.046009 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:58.046051 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":50,"threads_started":1}
I20260928 02:36:58.785710 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:58.785492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.1:55069)
0928 02:36:58.785616 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:58.785664 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
I20260928 02:36:58.786031 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:58.785836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.3:50655)
0928 02:36:58.785940 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:58.785980 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":57,"threads_started":1}
I20260928 02:36:59.046674 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:59.046477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.2:55087)
0928 02:36:59.046591 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:59.046627 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
I20260928 02:36:59.786389 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:59.786129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.1:32871)
0928 02:36:59.786276 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:59.786320 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
I20260928 02:36:59.786553 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:36:59.786382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.3:54205)
0928 02:36:59.786475 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 02:36:59.786506 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":50,"threads_started":1}
I20260928 02:37:00.047271 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:00.047044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.2:39295)
0928 02:37:00.047188 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:00.047225 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260928 02:37:00.787110 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:00.786901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.1:39723)
0928 02:37:00.787033 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:00.787064 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
I20260928 02:37:00.787139 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:00.786956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.3:34265)
0928 02:37:00.787066 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:00.787096 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":56,"threads_started":1}
I20260928 02:37:00.787439 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:00.787348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.1:39891)
0928 02:37:00.787394 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:00.787417 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 02:37:01.047896 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:01.047693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.2:42631)
0928 02:37:01.047806 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:01.047849 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260928 02:37:01.787730 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:01.787525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.3:55381)
0928 02:37:01.787649 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:01.787685 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20260928 02:37:01.787940 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:01.787746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.1:38725)
0928 02:37:01.787840 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:01.787893 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":43,"threads_started":1}
I20260928 02:37:02.048477 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:02.048268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.2:53277)
0928 02:37:02.048397 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:02.048432 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
I20260928 02:37:02.788388 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:02.788153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.3:44367)
0928 02:37:02.788308 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:02.788344 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":78,"threads_started":1}
I20260928 02:37:02.788511 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:02.788317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.1:36575)
0928 02:37:02.788428 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:02.788462 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":51,"threads_started":1}
I20260928 02:37:03.049070 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:03.048869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.2:53065)
0928 02:37:03.048984 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:03.049023 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20260928 02:37:03.789064 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:03.788851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.3:45767)
0928 02:37:03.788989 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:03.789020 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20260928 02:37:03.789093 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:03.788898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.1:45439)
0928 02:37:03.789026 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:03.789055 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
I20260928 02:37:04.049696 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:04.049469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.2:38661)
0928 02:37:04.049610 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:04.049646 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
I20260928 02:37:04.789664 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:04.789469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.3:53571)
0928 02:37:04.789587 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:04.789619 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260928 02:37:04.789747 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:04.789548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.1:42793)
0928 02:37:04.789668 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:04.789701 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
I20260928 02:37:05.050347 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:05.050141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.2:43393)
0928 02:37:05.050260 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:05.050298 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260928 02:37:05.790333 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:05.790101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.3:45641)
0928 02:37:05.790239 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:05.790287 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":58,"threads_started":1}
I20260928 02:37:05.790352 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:05.790168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.1:46633)
0928 02:37:05.790279 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:05.790312 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":62,"threads_started":1}
I20260928 02:37:06.050956 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:06.050766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.2:51839)
0928 02:37:06.050878 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:06.050908 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260928 02:37:06.790937 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:06.790731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.3:47155)
0928 02:37:06.790851 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:06.790884 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
I20260928 02:37:06.790953 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:06.790732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.1:53715)
0928 02:37:06.790873 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:06.790900 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":70,"threads_started":1}
I20260928 02:37:07.051589 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:07.051350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.2:35799)
0928 02:37:07.051498 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:07.051542 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":59,"threads_started":1}
I20260928 02:37:07.791611 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:07.791408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.1:52879)
0928 02:37:07.791539 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:07.791571 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":54,"threads_started":1}
I20260928 02:37:07.791610 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:07.791397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.3:59603)
0928 02:37:07.791526 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:07.791564 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260928 02:37:08.052316 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:08.052060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.2:59127)
0928 02:37:08.052211 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:08.052266 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
I20260928 02:37:08.792337 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:08.792080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.1:52793)
0928 02:37:08.792236 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:08.792288 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":70,"threads_started":1}
I20260928 02:37:08.792335 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:08.792080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.3:58269)
0928 02:37:08.792235 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:08.792276 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":72,"threads_started":1}
I20260928 02:37:09.053021 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:09.052805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.2:45133)
0928 02:37:09.052932 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:09.052975 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20260928 02:37:09.792972 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:09.792766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.1:52927)
0928 02:37:09.792892 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:09.792926 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260928 02:37:09.792976 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:09.792777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.3:46911)
0928 02:37:09.792903 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:09.792936 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":57,"threads_started":1}
I20260928 02:37:10.053649 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:10.053430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33383 (local address 127.0.98.2:53709)
0928 02:37:10.053553 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:10.053591 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20260928 02:37:10.533219 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 15219
I20260928 02:37:10.537707 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 15383
I20260928 02:37:10.541724 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 15518
2026-09-28T02:37:10Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18456 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:37:10 dist-test-slave-fz47 krb5kdc[15715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:37:10 dist-test-slave-fz47 krb5kdc[15715](info): set up 2 sockets
Sep 28 02:37:10 dist-test-slave-fz47 krb5kdc[15715](info): commencing operation
krb5kdc: starting...
W20260928 02:37:11.497012 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:37:11 dist-test-slave-fz47 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:37:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:37:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 02:37:11.606112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35419
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:37:11.683367 15728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:37:11.683561 15728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:37:11.683581 15728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:37:11.684634 15728 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 02:37:11.684664 15728 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 02:37:11.684675 15728 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 02:37:11.684685 15728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:37:11.684696 15728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:37:11.684715 15728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:37:11.684729 15728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:37:11.684746 15728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:37:11.686160 15728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:37:11.686345 15728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:37:11.686512 15728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:37:11.691063 15728 server_base.cc:1034] running on GCE node
W20260928 02:37:11.691161 15736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:37:11.691179 15734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:37:11.691306 15733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:37:11.691530 15728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:37:11.691785 15728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:37:11.692922 15728 hybrid_clock.cc:648] HybridClock initialized: now 1790563031692906 us; error 33 us; skew 500 ppm
Sep 28 02:37:11 dist-test-slave-fz47 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:37:11.695248 15728 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:37:11.695643 15728 webserver.cc:460] Webserver started at http://127.0.98.62:43591/ using document root <none> and password file <none>
I20260928 02:37:11.695825 15728 fs_manager.cc:359] Metadata directory not provided
I20260928 02:37:11.695866 15728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:37:11.695976 15728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:37:11.696951 15728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "f4550d46bc52494b98423de6c092eda5"
format_stamp: "Formatted at 2026-09-28 02:37:11 on dist-test-slave-fz47"
server_key: "dfa125bd31d63d79cb520a3cb79e5d1b"
server_key_iv: "333db4266a747246188820117fa19827"
server_key_version: "encryptionkey@0"
I20260928 02:37:11.697299 15728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "f4550d46bc52494b98423de6c092eda5"
format_stamp: "Formatted at 2026-09-28 02:37:11 on dist-test-slave-fz47"
server_key: "dfa125bd31d63d79cb520a3cb79e5d1b"
server_key_iv: "333db4266a747246188820117fa19827"
server_key_version: "encryptionkey@0"
I20260928 02:37:11.698459 15728 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:37:11.699031 15743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:11.699165 15728 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:37:11.699244 15728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "f4550d46bc52494b98423de6c092eda5"
format_stamp: "Formatted at 2026-09-28 02:37:11 on dist-test-slave-fz47"
server_key: "dfa125bd31d63d79cb520a3cb79e5d1b"
server_key_iv: "333db4266a747246188820117fa19827"
server_key_version: "encryptionkey@0"
I20260928 02:37:11.699316 15728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:37:11.720739 15728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:37:11.722608 15728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:37:11.722714 15728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:37:11.726279 15728 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35419
I20260928 02:37:11.726311 15795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35419 every 8 connection(s)
I20260928 02:37:11.726619 15728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:37:11.727401 15796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:37:11.729560 15796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5: Bootstrap starting.
I20260928 02:37:11.730098 15796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5: Neither blocks nor log segments found. Creating new log.
I20260928 02:37:11.730360 15796 log.cc:824] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:37:11.730536 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 15728
I20260928 02:37:11.730671 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:37:11.730885 392 external_mini_cluster.cc:1428] Setting key f58b0f971bfc1753e17820169db47731
I20260928 02:37:11.731721 15796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5: No bootstrap required, opened a new log
I20260928 02:37:11.734073 15796 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4550d46bc52494b98423de6c092eda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35419 } }
I20260928 02:37:11.734236 15796 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:37:11.734269 15796 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4550d46bc52494b98423de6c092eda5, State: Initialized, Role: FOLLOWER
I20260928 02:37:11.734390 15796 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4550d46bc52494b98423de6c092eda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35419 } }
I20260928 02:37:11.734462 15796 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:37:11.734495 15796 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:37:11.734535 15796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:37:11.735100 15796 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4550d46bc52494b98423de6c092eda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35419 } }
I20260928 02:37:11.735188 15796 leader_election.cc:302] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4550d46bc52494b98423de6c092eda5; no voters:
I20260928 02:37:11.735327 15796 leader_election.cc:288] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:37:11.735363 15801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:37:11.735507 15801 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [term 1 LEADER]: Becoming Leader. State: Replica: f4550d46bc52494b98423de6c092eda5, State: Running, Role: LEADER
I20260928 02:37:11.735587 15796 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:37:11.735636 15801 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4550d46bc52494b98423de6c092eda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35419 } }
I20260928 02:37:11.735921 15801 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4550d46bc52494b98423de6c092eda5. Latest consensus state: current_term: 1 leader_uuid: "f4550d46bc52494b98423de6c092eda5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4550d46bc52494b98423de6c092eda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35419 } } }
I20260928 02:37:11.735975 15803 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4550d46bc52494b98423de6c092eda5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4550d46bc52494b98423de6c092eda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35419 } } }
I20260928 02:37:11.736186 15803 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [sys.catalog]: This master's current role is: LEADER
I20260928 02:37:11.736155 15801 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [sys.catalog]: This master's current role is: LEADER
I20260928 02:37:11.736459 15810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:37:11.736927 15810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 02:37:11 dist-test-slave-fz47 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:37:11.740427 15810 catalog_manager.cc:1347] Generated new cluster ID: e72c588e73ad413da116559d40605921
I20260928 02:37:11.740471 15810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:37:11.741631 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:11.731321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40950 (local address 127.0.98.62:35419)
0928 02:37:11.731484 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 02:37:11.731491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:37:11.731707 (+ 216us) server_negotiation.cc:407] Connection header received
0928 02:37:11.732168 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:37:11.732174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:37:11.732220 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:37:11.732328 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:37:11.733031 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:11.733923 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:11.734899 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:11.735332 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:11.735705 (+ 373us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:11.738052 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:37:11.738056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:37:11.738056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:37:11.738068 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:37:11.739985 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:37:11.740317 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:37:11.740319 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:37:11.740320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:37:11.740403 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:37:11.740584 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:37:11.740585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:37:11.740585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:37:11.740944 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:37:11.741100 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 02:37:11.741321 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 02:37:11.741438 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
I20260928 02:37:11.748567 15810 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:37:11.749274 15810 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:37:11.762493 15810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5: Generated new TSK 0
I20260928 02:37:11.762672 15810 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 02:37:11.791656 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35419
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:37:11.870788 15822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:37:11.870983 15822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:37:11.871004 15822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:37:11.872090 15822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:37:11.872124 15822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:37:11.872148 15822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:37:11.873554 15822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:37:11.873728 15822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:37:11.873898 15822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:37:11.878504 15830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:37:11.878681 15822 server_base.cc:1034] running on GCE node
W20260928 02:37:11.878547 15827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:37:11.878504 15828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:37:11.878975 15822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:37:11.879213 15822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:37:11.880378 15822 hybrid_clock.cc:648] HybridClock initialized: now 1790563031880355 us; error 32 us; skew 500 ppm
Sep 28 02:37:11 dist-test-slave-fz47 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:37:11.882774 15822 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 02:37:11.883199 15822 webserver.cc:460] Webserver started at http://127.0.98.1:43739/ using document root <none> and password file <none>
I20260928 02:37:11.883401 15822 fs_manager.cc:359] Metadata directory not provided
I20260928 02:37:11.883448 15822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:37:11.883575 15822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:37:11.884577 15822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/data/instance:
uuid: "d5f2781c1daf49dab4f6ee269fb52c16"
format_stamp: "Formatted at 2026-09-28 02:37:11 on dist-test-slave-fz47"
server_key: "272e473f1948e79243f152b942d38319"
server_key_iv: "3f18022403f17ff9c653e7d8c5e3787e"
server_key_version: "encryptionkey@0"
I20260928 02:37:11.884894 15822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d5f2781c1daf49dab4f6ee269fb52c16"
format_stamp: "Formatted at 2026-09-28 02:37:11 on dist-test-slave-fz47"
server_key: "272e473f1948e79243f152b942d38319"
server_key_iv: "3f18022403f17ff9c653e7d8c5e3787e"
server_key_version: "encryptionkey@0"
I20260928 02:37:11.886194 15822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:37:11.886868 15837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:11.887022 15822 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:37:11.887099 15822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "d5f2781c1daf49dab4f6ee269fb52c16"
format_stamp: "Formatted at 2026-09-28 02:37:11 on dist-test-slave-fz47"
server_key: "272e473f1948e79243f152b942d38319"
server_key_iv: "3f18022403f17ff9c653e7d8c5e3787e"
server_key_version: "encryptionkey@0"
I20260928 02:37:11.887178 15822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:37:11.931385 15822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:37:11.933398 15822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:37:11.933530 15822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:37:11.933751 15822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:37:11.934031 15822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:37:11.934062 15822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:11.934090 15822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:37:11.934123 15822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:11.939513 15822 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45601
I20260928 02:37:11.939627 15950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45601 every 8 connection(s)
I20260928 02:37:11.939858 15822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 02:37:11 dist-test-slave-fz47 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:37:11.946436 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 15822
I20260928 02:37:11.946548 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/wal/instance
I20260928 02:37:11.946777 392 external_mini_cluster.cc:1428] Setting key 0d046d153362cdb869db789368f9a933
I20260928 02:37:11.948525 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:11.940603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51487 (local address 127.0.98.62:35419)
0928 02:37:11.940682 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 02:37:11.940684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:37:11.940765 (+ 81us) server_negotiation.cc:407] Connection header received
0928 02:37:11.941232 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:37:11.941233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:37:11.941241 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:37:11.941342 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:37:11.942235 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:11.942728 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:11.943635 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:11.943829 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:11.943872 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:11.945913 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:37:11.945914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:37:11.945915 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:37:11.945921 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:37:11.947232 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:37:11.947637 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:37:11.947637 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:37:11.947638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:37:11.947694 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:37:11.947910 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:37:11.947911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:37:11.947911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:37:11.948033 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:37:11.948091 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:37:11.948365 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 02:37:11.948441 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:37:11.948681 15953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:11.940499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.1:51487)
0928 02:37:11.940654 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:11.940695 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:37:11.941157 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:37:11.941396 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:37:11.941400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:37:11.941941 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:37:11.942163 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:11.942168 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:11.942786 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:11.942787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:11.943574 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:11.943576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:11.943915 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:11.943916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:11.944032 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:11.944466 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:37:11.944474 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:37:11.945823 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:37:11.947303 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:37:11.947310 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:37:11.947312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:37:11.947570 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:37:11.947762 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:37:11.947763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:37:11.947763 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:37:11.947859 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:37:11.948110 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:37:11.948117 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:37:11.948238 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 02:37:11.948336 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 02:37:11.948461 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
I20260928 02:37:11.949095 15951 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35419
I20260928 02:37:11.949177 15951 heartbeater.cc:464] Registering TS with master...
I20260928 02:37:11.949371 15951 heartbeater.cc:511] Master 127.0.98.62:35419 requested a full tablet report, sending...
I20260928 02:37:11.949893 15760 ts_manager.cc:194] Registered new tserver with Master: d5f2781c1daf49dab4f6ee269fb52c16 (127.0.98.1:45601)
I20260928 02:37:11.950485 15760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51487
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 02:37:11.968350 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35419
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:37:12.045226 15956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:37:12.045454 15956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:37:12.045483 15956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:37:12.046610 15956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:37:12.046650 15956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:37:12.046684 15956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:37:12.048079 15956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:37:12.048300 15956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:37:12.048492 15956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:37:12.053038 15964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:37:12.053043 15961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:37:12.053136 15956 server_base.cc:1034] running on GCE node
W20260928 02:37:12.053093 15962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:37:12.053393 15956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:37:12.053601 15956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:37:12.054747 15956 hybrid_clock.cc:648] HybridClock initialized: now 1790563032054735 us; error 23 us; skew 500 ppm
Sep 28 02:37:12 dist-test-slave-fz47 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:37:12.057093 15956 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 02:37:12.057515 15956 webserver.cc:460] Webserver started at http://127.0.98.2:43899/ using document root <none> and password file <none>
I20260928 02:37:12.057713 15956 fs_manager.cc:359] Metadata directory not provided
I20260928 02:37:12.057758 15956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:37:12.057873 15956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:37:12.058823 15956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/data/instance:
uuid: "3d4d5723230048c292ecfa086a24fad7"
format_stamp: "Formatted at 2026-09-28 02:37:12 on dist-test-slave-fz47"
server_key: "9cf975d98b8fe32d936fc3755746fa9e"
server_key_iv: "8b92e6ae33b0600db87f9badd28b87f2"
server_key_version: "encryptionkey@0"
I20260928 02:37:12.059136 15956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3d4d5723230048c292ecfa086a24fad7"
format_stamp: "Formatted at 2026-09-28 02:37:12 on dist-test-slave-fz47"
server_key: "9cf975d98b8fe32d936fc3755746fa9e"
server_key_iv: "8b92e6ae33b0600db87f9badd28b87f2"
server_key_version: "encryptionkey@0"
I20260928 02:37:12.060339 15956 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:37:12.061002 15971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:12.061146 15956 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:37:12.061219 15956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "3d4d5723230048c292ecfa086a24fad7"
format_stamp: "Formatted at 2026-09-28 02:37:12 on dist-test-slave-fz47"
server_key: "9cf975d98b8fe32d936fc3755746fa9e"
server_key_iv: "8b92e6ae33b0600db87f9badd28b87f2"
server_key_version: "encryptionkey@0"
I20260928 02:37:12.061304 15956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:37:12.083132 15956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:37:12.085141 15956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:37:12.085281 15956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:37:12.085479 15956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:37:12.085719 15956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:37:12.085753 15956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:12.085774 15956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:37:12.085784 15956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:12.091321 15956 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42391
I20260928 02:37:12.091363 16084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42391 every 8 connection(s)
I20260928 02:37:12.091635 15956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
I20260928 02:37:12.092490 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 15956
I20260928 02:37:12.092595 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/wal/instance
I20260928 02:37:12.092784 392 external_mini_cluster.cc:1428] Setting key b6d35ff3a1a5c907b945e95f7d6cd0b4
Sep 28 02:37:12 dist-test-slave-fz47 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 02:37:12.101576 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:12.092260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42371 (local address 127.0.98.62:35419)
0928 02:37:12.092309 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 02:37:12.092311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:37:12.092541 (+ 230us) server_negotiation.cc:407] Connection header received
0928 02:37:12.093811 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:37:12.093812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:37:12.093821 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:37:12.093854 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:37:12.094597 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:12.095082 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:12.096440 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:12.096636 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:12.096687 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:12.098955 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:37:12.098956 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:37:12.098957 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:37:12.098964 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:37:12.100189 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:37:12.100638 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:37:12.100639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:37:12.100639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:37:12.100685 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:37:12.100943 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:37:12.100943 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:37:12.100943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:37:12.101055 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:37:12.101102 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 02:37:12.101410 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 02:37:12.101493 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 02:37:12.101691 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:12.092227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.2:42371)
0928 02:37:12.092401 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:12.092448 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:37:12.093722 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:37:12.093930 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:37:12.093934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:37:12.094258 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:37:12.094517 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:12.094523 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:12.095162 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:12.095163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:12.096366 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:12.096370 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:12.096708 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:12.096709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:12.096853 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:12.097319 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:37:12.097328 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:37:12.098868 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:37:12.100264 (+ 1396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:37:12.100271 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:37:12.100274 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:37:12.100572 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:37:12.100742 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:37:12.100743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:37:12.100743 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:37:12.100873 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:37:12.101102 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:37:12.101108 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:37:12.101289 (+ 181us) client_negotiation.cc:769] Sending connection context
0928 02:37:12.101386 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 02:37:12.101504 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":74,"threads_started":1}
I20260928 02:37:12.102111 16085 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35419
I20260928 02:37:12.102202 16085 heartbeater.cc:464] Registering TS with master...
I20260928 02:37:12.102342 16085 heartbeater.cc:511] Master 127.0.98.62:35419 requested a full tablet report, sending...
I20260928 02:37:12.102720 15760 ts_manager.cc:194] Registered new tserver with Master: 3d4d5723230048c292ecfa086a24fad7 (127.0.98.2:42391)
I20260928 02:37:12.103078 15760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 02:37:12.115978 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35419
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:37:12.197556 16090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:37:12.197742 16090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:37:12.197763 16090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:37:12.198920 16090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:37:12.198959 16090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:37:12.198984 16090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:37:12.200389 16090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:37:12.200626 16090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:37:12.200798 16090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:37:12.205487 16098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:37:12.205513 16095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:37:12.205534 16096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:37:12.205729 16090 server_base.cc:1034] running on GCE node
I20260928 02:37:12.205935 16090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:37:12.206140 16090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:37:12.207285 16090 hybrid_clock.cc:648] HybridClock initialized: now 1790563032207266 us; error 34 us; skew 500 ppm
Sep 28 02:37:12 dist-test-slave-fz47 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:37:12.209493 16090 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 02:37:12.209899 16090 webserver.cc:460] Webserver started at http://127.0.98.3:43385/ using document root <none> and password file <none>
I20260928 02:37:12.210103 16090 fs_manager.cc:359] Metadata directory not provided
I20260928 02:37:12.210150 16090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:37:12.210265 16090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:37:12.211467 16090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/data/instance:
uuid: "8c164ae459ed4011874be6e0ea85ecad"
format_stamp: "Formatted at 2026-09-28 02:37:12 on dist-test-slave-fz47"
server_key: "60535341bc26bb63150fa3e3ce09e069"
server_key_iv: "5e63be34f400282bc75e900160eb40b0"
server_key_version: "encryptionkey@0"
I20260928 02:37:12.211822 16090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8c164ae459ed4011874be6e0ea85ecad"
format_stamp: "Formatted at 2026-09-28 02:37:12 on dist-test-slave-fz47"
server_key: "60535341bc26bb63150fa3e3ce09e069"
server_key_iv: "5e63be34f400282bc75e900160eb40b0"
server_key_version: "encryptionkey@0"
I20260928 02:37:12.212975 16090 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:37:12.213672 16105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:12.213811 16090 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:12.213896 16090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "8c164ae459ed4011874be6e0ea85ecad"
format_stamp: "Formatted at 2026-09-28 02:37:12 on dist-test-slave-fz47"
server_key: "60535341bc26bb63150fa3e3ce09e069"
server_key_iv: "5e63be34f400282bc75e900160eb40b0"
server_key_version: "encryptionkey@0"
I20260928 02:37:12.213959 16090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:37:12.223261 16090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:37:12.225092 16090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:37:12.225206 16090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:37:12.225426 16090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:37:12.225715 16090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:37:12.225749 16090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:12.225778 16090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:37:12.225802 16090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:12.230969 16090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45175
I20260928 02:37:12.231000 16218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45175 every 8 connection(s)
I20260928 02:37:12.231287 16090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
I20260928 02:37:12.231818 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 16090
I20260928 02:37:12.231907 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/wal/instance
I20260928 02:37:12.232101 392 external_mini_cluster.cc:1428] Setting key 4a79796b960c91493f2589c9e423ca43
Sep 28 02:37:12 dist-test-slave-fz47 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:37:12.239713 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:12.231962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39723 (local address 127.0.98.62:35419)
0928 02:37:12.232010 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 02:37:12.232011 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:37:12.232217 (+ 206us) server_negotiation.cc:407] Connection header received
0928 02:37:12.232751 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:37:12.232752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:37:12.232764 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:37:12.232812 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:37:12.233735 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:12.234220 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:12.235132 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:12.235334 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:12.235388 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:12.237343 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:37:12.237344 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:37:12.237345 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:37:12.237352 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:37:12.238529 (+ 1177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:37:12.238881 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:37:12.238883 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:37:12.238883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:37:12.238939 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:37:12.239142 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:37:12.239143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:37:12.239143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:37:12.239256 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:37:12.239310 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:37:12.239550 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 02:37:12.239628 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 02:37:12.239866 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:12.231918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.3:39723)
0928 02:37:12.232086 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:12.232130 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:37:12.232656 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:37:12.232878 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:37:12.232881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:37:12.233468 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:37:12.233645 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:12.233650 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:12.234297 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:12.234298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:12.235055 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:12.235057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:12.235398 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:12.235399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:12.235567 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:12.235978 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:37:12.235986 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:37:12.237234 (+ 1248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:37:12.238604 (+ 1370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:37:12.238611 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:37:12.238612 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:37:12.238818 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:37:12.238993 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:37:12.238993 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:37:12.238994 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:37:12.239089 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:37:12.239308 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:37:12.239314 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:37:12.239440 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 02:37:12.239524 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 02:37:12.239668 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
I20260928 02:37:12.240245 16219 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35419
I20260928 02:37:12.240320 16219 heartbeater.cc:464] Registering TS with master...
I20260928 02:37:12.240478 16219 heartbeater.cc:511] Master 127.0.98.62:35419 requested a full tablet report, sending...
I20260928 02:37:12.240811 15760 ts_manager.cc:194] Registered new tserver with Master: 8c164ae459ed4011874be6e0ea85ecad (127.0.98.3:45175)
I20260928 02:37:12.241139 15760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39723
I20260928 02:37:12.243582 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:37:12.249583 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:12.244764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40952 (local address 127.0.98.62:35419)
0928 02:37:12.244803 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:37:12.244804 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:37:12.244862 (+ 58us) server_negotiation.cc:407] Connection header received
0928 02:37:12.244903 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:37:12.244904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:37:12.244911 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:37:12.244954 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:37:12.245486 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:12.245940 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:12.246605 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:12.246794 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:12.246841 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:12.247754 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:37:12.247755 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:37:12.247756 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:37:12.247761 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:37:12.248761 (+ 1000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:37:12.249019 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:37:12.249020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:37:12.249020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:37:12.249059 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:37:12.249171 (+ 112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:37:12.249172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:37:12.249172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:37:12.249326 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:37:12.249396 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:37:12.249471 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 02:37:12.249522 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:37:12.250873 15760 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:37:12.252348 15760 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:37:12.258306 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:12.254193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45601 (local address 127.0.0.1:37160)
0928 02:37:12.254298 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:12.254334 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:37:12.254395 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:37:12.254816 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:37:12.254819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:37:12.254821 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:37:12.254882 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:12.254884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:12.256275 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:12.256276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:12.257121 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:12.257124 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:12.258012 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:12.258013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:12.258089 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:12.258093 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:37:12.258129 (+ 36us) client_negotiation.cc:240] Negotiation successful
0928 02:37:12.258209 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":28,"thread_start_us":63,"threads_started":1}
I20260928 02:37:12.258368 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:12.254372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37160 (local address 127.0.98.1:45601)
0928 02:37:12.254596 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 02:37:12.254601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:37:12.254613 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:37:12.254655 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:37:12.254657 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:37:12.254673 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:37:12.254797 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:37:12.255165 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:12.256190 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:12.257178 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:12.257912 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:12.258173 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:12.258233 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 02:37:12.258258 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 02:37:12.258293 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":79,"threads_started":1}
I20260928 02:37:12.258374 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:12.254632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47354 (local address 127.0.98.2:42391)
0928 02:37:12.254812 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 02:37:12.254815 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:37:12.254824 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:37:12.254858 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:37:12.254860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:37:12.254873 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:37:12.254946 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:37:12.255071 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:12.255794 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:12.257366 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:12.257871 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:12.257936 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:12.258003 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:37:12.258258 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 02:37:12.258298 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":74,"threads_started":1}
I20260928 02:37:12.258705 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:12.254198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42391 (local address 127.0.0.1:47354)
0928 02:37:12.254529 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:12.254539 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:37:12.254584 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:37:12.254956 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:37:12.254957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:37:12.254960 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:37:12.255017 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:12.255019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:12.255887 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:12.255888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:12.257241 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:12.257243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:12.257944 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:12.257945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:12.258040 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:12.258055 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 02:37:12.258121 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 02:37:12.258186 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":22,"thread_start_us":69,"threads_started":1}
I20260928 02:37:12.258767 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:12.254417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33840 (local address 127.0.98.3:45175)
0928 02:37:12.254615 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 02:37:12.254619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:37:12.254681 (+ 62us) server_negotiation.cc:407] Connection header received
0928 02:37:12.254740 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:37:12.254742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:37:12.254755 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:37:12.254820 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:37:12.254995 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:12.256065 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:12.257354 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:12.258023 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:12.258081 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:12.258135 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:37:12.258676 (+ 541us) server_negotiation.cc:299] Negotiation successful
0928 02:37:12.258709 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":101,"threads_started":1}
I20260928 02:37:12.258291 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:12.254198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45175 (local address 127.0.0.1:33840)
0928 02:37:12.254640 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:12.254647 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 02:37:12.254686 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:37:12.254889 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:37:12.254889 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:37:12.254891 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:37:12.254927 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:12.254928 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:12.256161 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:12.256163 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:12.257232 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:12.257234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:12.258083 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:12.258083 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:12.258136 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:12.258140 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:37:12.258181 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 02:37:12.258217 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":2,"thread_start_us":70,"threads_started":1}
I20260928 02:37:12.259081 15885 tablet_service.cc:1467] Processing CreateTablet for tablet 29c2dfaf9edf4ac3b9477caae7b7624f (DEFAULT_TABLE table=test-table [id=4e362469b30e40d790d0960b0a58bc33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:37:12.259457 15885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29c2dfaf9edf4ac3b9477caae7b7624f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:37:12.259589 16153 tablet_service.cc:1467] Processing CreateTablet for tablet 29c2dfaf9edf4ac3b9477caae7b7624f (DEFAULT_TABLE table=test-table [id=4e362469b30e40d790d0960b0a58bc33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:37:12.259876 16153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29c2dfaf9edf4ac3b9477caae7b7624f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:37:12.261724 16019 tablet_service.cc:1467] Processing CreateTablet for tablet 29c2dfaf9edf4ac3b9477caae7b7624f (DEFAULT_TABLE table=test-table [id=4e362469b30e40d790d0960b0a58bc33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:37:12.262049 16019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29c2dfaf9edf4ac3b9477caae7b7624f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:37:12.262045 16233 tablet_bootstrap.cc:492] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16: Bootstrap starting.
I20260928 02:37:12.262569 16233 tablet_bootstrap.cc:654] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16: Neither blocks nor log segments found. Creating new log.
I20260928 02:37:12.262600 16234 tablet_bootstrap.cc:492] T 29c2dfaf9edf4ac3b9477caae7b7624f P 8c164ae459ed4011874be6e0ea85ecad: Bootstrap starting.
I20260928 02:37:12.263129 16233 log.cc:824] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16: Log is configured to *not* fsync() on all Append() calls
I20260928 02:37:12.263236 16234 tablet_bootstrap.cc:654] T 29c2dfaf9edf4ac3b9477caae7b7624f P 8c164ae459ed4011874be6e0ea85ecad: Neither blocks nor log segments found. Creating new log.
I20260928 02:37:12.263705 16234 log.cc:824] T 29c2dfaf9edf4ac3b9477caae7b7624f P 8c164ae459ed4011874be6e0ea85ecad: Log is configured to *not* fsync() on all Append() calls
I20260928 02:37:12.263876 16233 tablet_bootstrap.cc:492] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16: No bootstrap required, opened a new log
I20260928 02:37:12.263931 16233 ts_tablet_manager.cc:1397] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:37:12.264420 16234 tablet_bootstrap.cc:492] T 29c2dfaf9edf4ac3b9477caae7b7624f P 8c164ae459ed4011874be6e0ea85ecad: No bootstrap required, opened a new log
I20260928 02:37:12.264472 16234 ts_tablet_manager.cc:1397] T 29c2dfaf9edf4ac3b9477caae7b7624f P 8c164ae459ed4011874be6e0ea85ecad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:37:12.264948 16237 tablet_bootstrap.cc:492] T 29c2dfaf9edf4ac3b9477caae7b7624f P 3d4d5723230048c292ecfa086a24fad7: Bootstrap starting.
I20260928 02:37:12.265578 16237 tablet_bootstrap.cc:654] T 29c2dfaf9edf4ac3b9477caae7b7624f P 3d4d5723230048c292ecfa086a24fad7: Neither blocks nor log segments found. Creating new log.
I20260928 02:37:12.265928 16233 raft_consensus.cc:348] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45601 } } peers { permanent_uuid: "8c164ae459ed4011874be6e0ea85ecad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45175 } } peers { permanent_uuid: "3d4d5723230048c292ecfa086a24fad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } }
I20260928 02:37:12.266110 16233 raft_consensus.cc:374] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:37:12.266112 16237 log.cc:824] T 29c2dfaf9edf4ac3b9477caae7b7624f P 3d4d5723230048c292ecfa086a24fad7: Log is configured to *not* fsync() on all Append() calls
I20260928 02:37:12.266144 16233 raft_consensus.cc:729] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5f2781c1daf49dab4f6ee269fb52c16, State: Initialized, Role: FOLLOWER
I20260928 02:37:12.266274 16233 consensus_queue.cc:260] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45601 } } peers { permanent_uuid: "8c164ae459ed4011874be6e0ea85ecad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45175 } } peers { permanent_uuid: "3d4d5723230048c292ecfa086a24fad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } }
I20260928 02:37:12.266287 16234 raft_consensus.cc:348] T 29c2dfaf9edf4ac3b9477caae7b7624f P 8c164ae459ed4011874be6e0ea85ecad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45601 } } peers { permanent_uuid: "8c164ae459ed4011874be6e0ea85ecad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45175 } } peers { permanent_uuid: "3d4d5723230048c292ecfa086a24fad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } }
I20260928 02:37:12.266463 16234 raft_consensus.cc:374] T 29c2dfaf9edf4ac3b9477caae7b7624f P 8c164ae459ed4011874be6e0ea85ecad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:37:12.266513 16234 raft_consensus.cc:729] T 29c2dfaf9edf4ac3b9477caae7b7624f P 8c164ae459ed4011874be6e0ea85ecad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c164ae459ed4011874be6e0ea85ecad, State: Initialized, Role: FOLLOWER
I20260928 02:37:12.266541 16233 ts_tablet_manager.cc:1428] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:37:12.266661 15951 heartbeater.cc:503] Master 127.0.98.62:35419 was elected leader, sending a full tablet report...
I20260928 02:37:12.266671 16234 consensus_queue.cc:260] T 29c2dfaf9edf4ac3b9477caae7b7624f P 8c164ae459ed4011874be6e0ea85ecad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45601 } } peers { permanent_uuid: "8c164ae459ed4011874be6e0ea85ecad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45175 } } peers { permanent_uuid: "3d4d5723230048c292ecfa086a24fad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } }
I20260928 02:37:12.266824 16237 tablet_bootstrap.cc:492] T 29c2dfaf9edf4ac3b9477caae7b7624f P 3d4d5723230048c292ecfa086a24fad7: No bootstrap required, opened a new log
I20260928 02:37:12.266871 16237 ts_tablet_manager.cc:1397] T 29c2dfaf9edf4ac3b9477caae7b7624f P 3d4d5723230048c292ecfa086a24fad7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 02:37:12.266888 16234 ts_tablet_manager.cc:1428] T 29c2dfaf9edf4ac3b9477caae7b7624f P 8c164ae459ed4011874be6e0ea85ecad: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:37:12.266916 16219 heartbeater.cc:503] Master 127.0.98.62:35419 was elected leader, sending a full tablet report...
I20260928 02:37:12.268389 16237 raft_consensus.cc:348] T 29c2dfaf9edf4ac3b9477caae7b7624f P 3d4d5723230048c292ecfa086a24fad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45601 } } peers { permanent_uuid: "8c164ae459ed4011874be6e0ea85ecad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45175 } } peers { permanent_uuid: "3d4d5723230048c292ecfa086a24fad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } }
I20260928 02:37:12.268518 16237 raft_consensus.cc:374] T 29c2dfaf9edf4ac3b9477caae7b7624f P 3d4d5723230048c292ecfa086a24fad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:37:12.268548 16237 raft_consensus.cc:729] T 29c2dfaf9edf4ac3b9477caae7b7624f P 3d4d5723230048c292ecfa086a24fad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d4d5723230048c292ecfa086a24fad7, State: Initialized, Role: FOLLOWER
I20260928 02:37:12.268649 16237 consensus_queue.cc:260] T 29c2dfaf9edf4ac3b9477caae7b7624f P 3d4d5723230048c292ecfa086a24fad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45601 } } peers { permanent_uuid: "8c164ae459ed4011874be6e0ea85ecad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45175 } } peers { permanent_uuid: "3d4d5723230048c292ecfa086a24fad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } }
I20260928 02:37:12.268923 16085 heartbeater.cc:503] Master 127.0.98.62:35419 was elected leader, sending a full tablet report...
I20260928 02:37:12.268906 16237 ts_tablet_manager.cc:1428] T 29c2dfaf9edf4ac3b9477caae7b7624f P 3d4d5723230048c292ecfa086a24fad7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260928 02:37:12.342149 16086 tablet.cc:2369] T 29c2dfaf9edf4ac3b9477caae7b7624f P 3d4d5723230048c292ecfa086a24fad7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:37:12.410907 16239 raft_consensus.cc:482] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:37:12.411059 16239 raft_consensus.cc:504] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45601 } } peers { permanent_uuid: "8c164ae459ed4011874be6e0ea85ecad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45175 } } peers { permanent_uuid: "3d4d5723230048c292ecfa086a24fad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } }
I20260928 02:37:12.411410 16239 leader_election.cc:288] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c164ae459ed4011874be6e0ea85ecad (127.0.98.3:45175), 3d4d5723230048c292ecfa086a24fad7 (127.0.98.2:42391)
I20260928 02:37:12.414081 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:12.411649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39953 (local address 127.0.98.3:45175)
0928 02:37:12.411688 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:37:12.411690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:37:12.411699 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:37:12.411756 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:37:12.411757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:37:12.411768 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:37:12.411813 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:37:12.411940 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:12.412407 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:12.413363 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:12.413754 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:12.413820 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:12.413830 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 02:37:12.413966 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 02:37:12.414003 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:37:12.414067 15953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:12.411588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45175 (local address 127.0.98.1:39953)
0928 02:37:12.411641 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:12.411651 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:37:12.411702 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:37:12.411820 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:37:12.411821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:37:12.411823 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:37:12.411887 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:12.411888 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:12.412479 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:12.412480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:12.413290 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:12.413292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:12.413828 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:12.413829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:12.413902 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:12.413905 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:37:12.413955 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 02:37:12.413994 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":50}
I20260928 02:37:12.414155 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:12.411643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55593 (local address 127.0.98.2:42391)
0928 02:37:12.411689 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 02:37:12.411691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:37:12.411830 (+ 139us) server_negotiation.cc:407] Connection header received
0928 02:37:12.411879 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:37:12.411880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:37:12.411892 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:37:12.411940 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:37:12.412074 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:12.412531 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:12.413500 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:12.413856 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:12.413911 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:12.413920 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:37:12.414050 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 02:37:12.414088 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 02:37:12.414184 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:12.411588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42391 (local address 127.0.98.1:55593)
0928 02:37:12.411764 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:12.411772 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:37:12.411828 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:37:12.411947 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:37:12.411948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:37:12.411949 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:37:12.412022 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:12.412024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:12.412604 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:12.412605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:12.413421 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:12.413423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:12.413921 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:12.413922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:12.413992 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:12.413996 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:37:12.414040 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 02:37:12.414074 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":21,"thread_start_us":78,"threads_started":1}
I20260928 02:37:12.414458 16173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c2dfaf9edf4ac3b9477caae7b7624f" candidate_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c164ae459ed4011874be6e0ea85ecad" is_pre_election: true
I20260928 02:37:12.414474 16039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c2dfaf9edf4ac3b9477caae7b7624f" candidate_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d4d5723230048c292ecfa086a24fad7" is_pre_election: true
I20260928 02:37:12.414592 16173 raft_consensus.cc:2454] T 29c2dfaf9edf4ac3b9477caae7b7624f P 8c164ae459ed4011874be6e0ea85ecad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5f2781c1daf49dab4f6ee269fb52c16 in term 0.
I20260928 02:37:12.414611 16039 raft_consensus.cc:2454] T 29c2dfaf9edf4ac3b9477caae7b7624f P 3d4d5723230048c292ecfa086a24fad7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5f2781c1daf49dab4f6ee269fb52c16 in term 0.
I20260928 02:37:12.414805 15841 leader_election.cc:302] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c164ae459ed4011874be6e0ea85ecad, d5f2781c1daf49dab4f6ee269fb52c16; no voters:
I20260928 02:37:12.414932 16239 raft_consensus.cc:2781] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:37:12.414988 16239 raft_consensus.cc:482] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:37:12.415004 16239 raft_consensus.cc:3037] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:37:12.415622 16239 raft_consensus.cc:504] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45601 } } peers { permanent_uuid: "8c164ae459ed4011874be6e0ea85ecad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45175 } } peers { permanent_uuid: "3d4d5723230048c292ecfa086a24fad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } }
I20260928 02:37:12.415819 16239 leader_election.cc:288] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16 [CANDIDATE]: Term 1 election: Requested vote from peers 8c164ae459ed4011874be6e0ea85ecad (127.0.98.3:45175), 3d4d5723230048c292ecfa086a24fad7 (127.0.98.2:42391)
I20260928 02:37:12.415894 16173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c2dfaf9edf4ac3b9477caae7b7624f" candidate_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c164ae459ed4011874be6e0ea85ecad"
I20260928 02:37:12.415902 16039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29c2dfaf9edf4ac3b9477caae7b7624f" candidate_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d4d5723230048c292ecfa086a24fad7"
I20260928 02:37:12.415963 16173 raft_consensus.cc:3037] T 29c2dfaf9edf4ac3b9477caae7b7624f P 8c164ae459ed4011874be6e0ea85ecad [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:37:12.415977 16039 raft_consensus.cc:3037] T 29c2dfaf9edf4ac3b9477caae7b7624f P 3d4d5723230048c292ecfa086a24fad7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:37:12.416664 16173 raft_consensus.cc:2454] T 29c2dfaf9edf4ac3b9477caae7b7624f P 8c164ae459ed4011874be6e0ea85ecad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5f2781c1daf49dab4f6ee269fb52c16 in term 1.
I20260928 02:37:12.416719 16039 raft_consensus.cc:2454] T 29c2dfaf9edf4ac3b9477caae7b7624f P 3d4d5723230048c292ecfa086a24fad7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5f2781c1daf49dab4f6ee269fb52c16 in term 1.
I20260928 02:37:12.416817 15841 leader_election.cc:302] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c164ae459ed4011874be6e0ea85ecad, d5f2781c1daf49dab4f6ee269fb52c16; no voters:
I20260928 02:37:12.416898 16239 raft_consensus.cc:2781] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:37:12.417043 16239 raft_consensus.cc:686] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16 [term 1 LEADER]: Becoming Leader. State: Replica: d5f2781c1daf49dab4f6ee269fb52c16, State: Running, Role: LEADER
I20260928 02:37:12.417135 16239 consensus_queue.cc:237] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45601 } } peers { permanent_uuid: "8c164ae459ed4011874be6e0ea85ecad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45175 } } peers { permanent_uuid: "3d4d5723230048c292ecfa086a24fad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } }
I20260928 02:37:12.417922 15759 catalog_manager.cc:5495] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5f2781c1daf49dab4f6ee269fb52c16 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c164ae459ed4011874be6e0ea85ecad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d4d5723230048c292ecfa086a24fad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } health_report { overall_health: UNKNOWN } } }
I20260928 02:37:12.440121 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:12.438195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37174 (local address 127.0.98.1:45601)
0928 02:37:12.438244 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 02:37:12.438245 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:37:12.438255 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:37:12.438285 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:37:12.438286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:37:12.438303 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:37:12.438361 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:37:12.438458 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:12.438912 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:12.439572 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:12.439754 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:12.439793 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:12.439894 (+ 101us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:37:12.439981 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:37:12.440034 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:37:12.440050 (+ 16us) server_negotiation.cc:299] Negotiation successful
0928 02:37:12.440065 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260928 02:37:12.440464 15952 tablet.cc:2369] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:37:12.441704 16039 raft_consensus.cc:1261] T 29c2dfaf9edf4ac3b9477caae7b7624f P 3d4d5723230048c292ecfa086a24fad7 [term 1 FOLLOWER]: Refusing update from remote peer d5f2781c1daf49dab4f6ee269fb52c16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:37:12.441704 16173 raft_consensus.cc:1261] T 29c2dfaf9edf4ac3b9477caae7b7624f P 8c164ae459ed4011874be6e0ea85ecad [term 1 FOLLOWER]: Refusing update from remote peer d5f2781c1daf49dab4f6ee269fb52c16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:37:12.442003 16239 consensus_queue.cc:1035] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c164ae459ed4011874be6e0ea85ecad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:37:12.442132 16243 consensus_queue.cc:1035] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d4d5723230048c292ecfa086a24fad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:37:12.445305 16248 mvcc.cc:205] Tried to move back new op lower bound from 7334146180878950400 to 7334146180781346816. Current Snapshot: MvccSnapshot[applied={T|T < 7334146180878950400 or (T in {7334146180878950400})}]
I20260928 02:37:12.445433 16247 mvcc.cc:205] Tried to move back new op lower bound from 7334146180878950400 to 7334146180781346816. Current Snapshot: MvccSnapshot[applied={T|T < 7334146180878950400}]
I20260928 02:37:12.446760 15760 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:37:12.446836 15760 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:37:12.447479 15760 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5: Sending DeleteTablet for 3 replicas of tablet 29c2dfaf9edf4ac3b9477caae7b7624f
I20260928 02:37:12.447749 15885 tablet_service.cc:1514] Processing DeleteTablet for tablet 29c2dfaf9edf4ac3b9477caae7b7624f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:37:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37160
I20260928 02:37:12.447790 16153 tablet_service.cc:1514] Processing DeleteTablet for tablet 29c2dfaf9edf4ac3b9477caae7b7624f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:37:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33840
I20260928 02:37:12.447921 16019 tablet_service.cc:1514] Processing DeleteTablet for tablet 29c2dfaf9edf4ac3b9477caae7b7624f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:37:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47354
I20260928 02:37:12.447954 16256 tablet_replica.cc:330] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16: stopping tablet replica
I20260928 02:37:12.447996 16257 tablet_replica.cc:330] T 29c2dfaf9edf4ac3b9477caae7b7624f P 8c164ae459ed4011874be6e0ea85ecad: stopping tablet replica
I20260928 02:37:12.448033 16256 raft_consensus.cc:2229] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:37:12.448082 16257 raft_consensus.cc:2229] T 29c2dfaf9edf4ac3b9477caae7b7624f P 8c164ae459ed4011874be6e0ea85ecad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:37:12.448110 16256 raft_consensus.cc:2258] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:37:12.448191 16257 raft_consensus.cc:2258] T 29c2dfaf9edf4ac3b9477caae7b7624f P 8c164ae459ed4011874be6e0ea85ecad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:37:12.448238 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 15822
I20260928 02:37:12.448565 16257 ts_tablet_manager.cc:1905] T 29c2dfaf9edf4ac3b9477caae7b7624f P 8c164ae459ed4011874be6e0ea85ecad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:37:12.449880 16258 tablet_replica.cc:330] T 29c2dfaf9edf4ac3b9477caae7b7624f P 3d4d5723230048c292ecfa086a24fad7: stopping tablet replica
I20260928 02:37:12.449970 16258 raft_consensus.cc:2229] T 29c2dfaf9edf4ac3b9477caae7b7624f P 3d4d5723230048c292ecfa086a24fad7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:37:12.450062 16258 raft_consensus.cc:2258] T 29c2dfaf9edf4ac3b9477caae7b7624f P 3d4d5723230048c292ecfa086a24fad7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:37:12.450388 16258 ts_tablet_manager.cc:1905] T 29c2dfaf9edf4ac3b9477caae7b7624f P 3d4d5723230048c292ecfa086a24fad7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:37:12.451447 16258 ts_tablet_manager.cc:1918] T 29c2dfaf9edf4ac3b9477caae7b7624f P 3d4d5723230048c292ecfa086a24fad7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:37:12.451504 16258 log.cc:1196] T 29c2dfaf9edf4ac3b9477caae7b7624f P 3d4d5723230048c292ecfa086a24fad7: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/wal/wals/29c2dfaf9edf4ac3b9477caae7b7624f
I20260928 02:37:12.451700 16258 ts_tablet_manager.cc:1939] T 29c2dfaf9edf4ac3b9477caae7b7624f P 3d4d5723230048c292ecfa086a24fad7: Deleting consensus metadata
I20260928 02:37:12.452207 15745 catalog_manager.cc:4841] TS 3d4d5723230048c292ecfa086a24fad7 (127.0.98.2:42391): tablet 29c2dfaf9edf4ac3b9477caae7b7624f (table test-table [id=4e362469b30e40d790d0960b0a58bc33]) successfully deleted
I20260928 02:37:12.453349 16257 ts_tablet_manager.cc:1918] T 29c2dfaf9edf4ac3b9477caae7b7624f P 8c164ae459ed4011874be6e0ea85ecad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:37:12.453401 16257 log.cc:1196] T 29c2dfaf9edf4ac3b9477caae7b7624f P 8c164ae459ed4011874be6e0ea85ecad: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/wal/wals/29c2dfaf9edf4ac3b9477caae7b7624f
W20260928 02:37:12.453603 15744 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45601 (error 108)
I20260928 02:37:12.453635 16257 ts_tablet_manager.cc:1939] T 29c2dfaf9edf4ac3b9477caae7b7624f P 8c164ae459ed4011874be6e0ea85ecad: Deleting consensus metadata
I20260928 02:37:12.453960 15747 catalog_manager.cc:4841] TS 8c164ae459ed4011874be6e0ea85ecad (127.0.98.3:45175): tablet 29c2dfaf9edf4ac3b9477caae7b7624f (table test-table [id=4e362469b30e40d790d0960b0a58bc33]) successfully deleted
I20260928 02:37:12.454105 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 15956
I20260928 02:37:12.454125 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:12.454002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45601 (local address 127.0.0.1:37186)
0928 02:37:12.454053 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:12.454094 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 02:37:12.454839 15744 catalog_manager.cc:4584] TS d5f2781c1daf49dab4f6ee269fb52c16 (127.0.98.1:45601): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 29c2dfaf9edf4ac3b9477caae7b7624f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45601: connect: Connection refused (error 111)
I20260928 02:37:12.458510 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 16090
I20260928 02:37:12.462081 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 15728
I20260928 02:37:32.466470 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35419
--webserver_interface=127.0.98.62
--webserver_port=43591
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:37:32.543825 16260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:37:32.544018 16260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:37:32.544047 16260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:37:32.545187 16260 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 02:37:32.545222 16260 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 02:37:32.545240 16260 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 02:37:32.545276 16260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:37:32.545296 16260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:37:32.545317 16260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:37:32.545341 16260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:37:32.545358 16260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:37:32.546802 16260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:37:32.547035 16260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:37:32.547237 16260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:37:32.555233 16266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:37:32.555222 16269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:37:32.555222 16267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:37:32.555423 16260 server_base.cc:1034] running on GCE node
I20260928 02:37:32.555765 16260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:37:32.556080 16260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:37:32.557225 16260 hybrid_clock.cc:648] HybridClock initialized: now 1790563052557207 us; error 33 us; skew 500 ppm
Sep 28 02:37:32 dist-test-slave-fz47 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:37:32.559535 16260 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:37:32.559993 16260 webserver.cc:460] Webserver started at http://127.0.98.62:43591/ using document root <none> and password file <none>
I20260928 02:37:32.560216 16260 fs_manager.cc:359] Metadata directory not provided
I20260928 02:37:32.560276 16260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:37:32.561565 16260 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:37:32.562125 16276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:32.562264 16260 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:37:32.562343 16260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "f4550d46bc52494b98423de6c092eda5"
format_stamp: "Formatted at 2026-09-28 02:37:11 on dist-test-slave-fz47"
server_key: "dfa125bd31d63d79cb520a3cb79e5d1b"
server_key_iv: "333db4266a747246188820117fa19827"
server_key_version: "encryptionkey@0"
I20260928 02:37:32.562722 16260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:37:32.587568 16260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:37:32.589501 16260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:37:32.589629 16260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:37:32.593218 16260 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35419
I20260928 02:37:32.593276 16328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35419 every 8 connection(s)
I20260928 02:37:32.593626 16260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:37:32.595014 16329 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:37:32.595660 16329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5: Bootstrap starting.
I20260928 02:37:32.601251 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 16260
I20260928 02:37:32.601611 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45601
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35419
--webserver_port=43739
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:37:32.604066 16329 log.cc:824] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:37:32.605995 16329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:37:32.606236 16329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5: Bootstrap complete.
I20260928 02:37:32.608304 16329 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4550d46bc52494b98423de6c092eda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35419 } }
I20260928 02:37:32.608577 16329 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4550d46bc52494b98423de6c092eda5, State: Initialized, Role: FOLLOWER
I20260928 02:37:32.608731 16329 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4550d46bc52494b98423de6c092eda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35419 } }
I20260928 02:37:32.608783 16329 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:37:32.608831 16329 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:37:32.608876 16329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:37:32.609927 16329 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4550d46bc52494b98423de6c092eda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35419 } }
I20260928 02:37:32.610013 16329 leader_election.cc:302] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4550d46bc52494b98423de6c092eda5; no voters:
I20260928 02:37:32.610162 16329 leader_election.cc:288] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:37:32.610213 16333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:37:32.610394 16333 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [term 2 LEADER]: Becoming Leader. State: Replica: f4550d46bc52494b98423de6c092eda5, State: Running, Role: LEADER
I20260928 02:37:32.610412 16329 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:37:32.610533 16333 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4550d46bc52494b98423de6c092eda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35419 } }
I20260928 02:37:32.610863 16334 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f4550d46bc52494b98423de6c092eda5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4550d46bc52494b98423de6c092eda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35419 } } }
I20260928 02:37:32.610878 16335 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4550d46bc52494b98423de6c092eda5. Latest consensus state: current_term: 2 leader_uuid: "f4550d46bc52494b98423de6c092eda5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4550d46bc52494b98423de6c092eda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35419 } } }
I20260928 02:37:32.610956 16334 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [sys.catalog]: This master's current role is: LEADER
I20260928 02:37:32.610976 16335 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [sys.catalog]: This master's current role is: LEADER
I20260928 02:37:32.611387 16341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:37:32.612160 16341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:37:32.612291 16341 catalog_manager.cc:1259] Loaded cluster ID: e72c588e73ad413da116559d40605921
I20260928 02:37:32.612521 16341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:37:32.613798 16341 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:37:32.614032 16341 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5: Loaded TSK: 0
I20260928 02:37:32.622531 16341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5: Generated new TSK 1
I20260928 02:37:32.622622 16341 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:37:32.683791 16331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:37:32.683980 16331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:37:32.684001 16331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:37:32.685112 16331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:37:32.685151 16331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:37:32.685175 16331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:37:32.686589 16331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43739
--tserver_master_addrs=127.0.98.62:35419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:37:32.686790 16331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:37:32.687000 16331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:37:32.692556 16357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:37:32.692684 16359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:37:32.692570 16331 server_base.cc:1034] running on GCE node
W20260928 02:37:32.692598 16356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:37:32.692977 16331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:37:32.693185 16331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:37:32.694316 16331 hybrid_clock.cc:648] HybridClock initialized: now 1790563052694303 us; error 29 us; skew 500 ppm
Sep 28 02:37:32 dist-test-slave-fz47 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:37:32.696552 16331 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 02:37:32.696936 16331 webserver.cc:460] Webserver started at http://127.0.98.1:43739/ using document root <none> and password file <none>
I20260928 02:37:32.697139 16331 fs_manager.cc:359] Metadata directory not provided
I20260928 02:37:32.697191 16331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:37:32.698414 16331 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:37:32.699003 16366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:32.699172 16331 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:37:32.699246 16331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "d5f2781c1daf49dab4f6ee269fb52c16"
format_stamp: "Formatted at 2026-09-28 02:37:11 on dist-test-slave-fz47"
server_key: "272e473f1948e79243f152b942d38319"
server_key_iv: "3f18022403f17ff9c653e7d8c5e3787e"
server_key_version: "encryptionkey@0"
I20260928 02:37:32.699579 16331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:37:32.714452 16331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:37:32.716398 16331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:37:32.716528 16331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:37:32.716727 16331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:37:32.717137 16373 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 02:37:32.718487 16331 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:37:32.718535 16331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 02:37:32.718557 16331 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:37:32.719137 16331 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:37:32.719177 16331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 02:37:32.719236 16373 tablet_bootstrap.cc:492] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16: Bootstrap starting.
I20260928 02:37:32.726074 16331 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45601
I20260928 02:37:32.726147 16480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45601 every 8 connection(s)
I20260928 02:37:32.726594 16331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
I20260928 02:37:32.726774 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 16331
I20260928 02:37:32.727063 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42391
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35419
--webserver_port=43899
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:37:32.729773 16373 log.cc:824] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16: Log is configured to *not* fsync() on all Append() calls
I20260928 02:37:32.731737 16373 tablet_bootstrap.cc:492] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:37:32.731985 16373 tablet_bootstrap.cc:492] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16: Bootstrap complete.
I20260928 02:37:32.732685 16373 ts_tablet_manager.cc:1397] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.002s
I20260928 02:37:32.733707 16373 raft_consensus.cc:348] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45601 } } peers { permanent_uuid: "8c164ae459ed4011874be6e0ea85ecad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45175 } } peers { permanent_uuid: "3d4d5723230048c292ecfa086a24fad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } }
I20260928 02:37:32.733927 16373 raft_consensus.cc:729] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5f2781c1daf49dab4f6ee269fb52c16, State: Initialized, Role: FOLLOWER
I20260928 02:37:32.734071 16373 consensus_queue.cc:260] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45601 } } peers { permanent_uuid: "8c164ae459ed4011874be6e0ea85ecad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45175 } } peers { permanent_uuid: "3d4d5723230048c292ecfa086a24fad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } }
I20260928 02:37:32.734310 16373 ts_tablet_manager.cc:1428] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
Sep 28 02:37:32 dist-test-slave-fz47 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:37:32.740646 16484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:32.728005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52087 (local address 127.0.98.62:35419)
0928 02:37:32.729361 (+ 1356us) server_negotiation.cc:206] Beginning negotiation
0928 02:37:32.729369 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:37:32.729404 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:37:32.729519 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:37:32.729524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:37:32.729573 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:37:32.729689 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:37:32.730502 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:32.731824 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:32.734553 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:32.735149 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:32.735263 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:32.737619 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:37:32.737623 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:37:32.737623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:37:32.737632 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:37:32.739049 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:37:32.739503 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:37:32.739505 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:37:32.739505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:37:32.739584 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:37:32.739838 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:37:32.739839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:37:32.739839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:37:32.740027 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:37:32.740081 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:37:32.740418 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 02:37:32.740515 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1309,"thread_start_us":89,"threads_started":1}
I20260928 02:37:32.740746 16483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:32.727655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.1:52087)
0928 02:37:32.728561 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:32.728606 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:37:32.729165 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:37:32.729809 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:37:32.729813 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:37:32.730147 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:37:32.730383 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:32.730389 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:32.733332 (+ 2943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:32.733334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:32.734467 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:32.734470 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:32.735246 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:32.735247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:32.735389 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:32.735946 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:37:32.735953 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:37:32.737524 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:37:32.739126 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:37:32.739133 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:37:32.739135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:37:32.739431 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:37:32.739643 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:37:32.739644 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:37:32.739644 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:37:32.739771 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:37:32.740085 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:37:32.740091 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:37:32.740258 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 02:37:32.740388 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 02:37:32.740552 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"thread_start_us":107,"threads_started":1}
I20260928 02:37:32.741446 16481 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35419
I20260928 02:37:32.741535 16481 heartbeater.cc:464] Registering TS with master...
I20260928 02:37:32.741693 16481 heartbeater.cc:511] Master 127.0.98.62:35419 requested a full tablet report, sending...
I20260928 02:37:32.742463 16293 ts_manager.cc:194] Registered new tserver with Master: d5f2781c1daf49dab4f6ee269fb52c16 (127.0.98.1:45601)
I20260928 02:37:32.743655 16293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52087
I20260928 02:37:32.744709 16481 heartbeater.cc:503] Master 127.0.98.62:35419 was elected leader, sending a full tablet report...
Sep 28 02:37:32 dist-test-slave-fz47 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 02:37:32.750415 16489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:32.743203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37236 (local address 127.0.98.1:45601)
0928 02:37:32.743353 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 02:37:32.743357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:37:32.743366 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:37:32.743395 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:37:32.743396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:37:32.743409 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:37:32.743467 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:37:32.743870 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:32.744655 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:32.745636 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:32.745919 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:32.745975 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:32.747925 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:37:32.747931 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:37:32.747932 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:37:32.747938 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:37:32.749350 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:37:32.749675 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:37:32.749678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:37:32.749678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:37:32.749732 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:37:32.749885 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:37:32.749886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:37:32.749886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:37:32.750017 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:37:32.750082 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 02:37:32.750238 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 02:37:32.750321 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20260928 02:37:32.750418 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:32.743013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45601 (local address 127.0.0.1:37236)
0928 02:37:32.743193 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:32.743227 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:37:32.743293 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:37:32.743480 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:37:32.743484 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:37:32.743739 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:37:32.743796 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:32.743799 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:32.744728 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:32.744729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:32.745566 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:32.745569 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:32.745981 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:32.745984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:32.746057 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:32.746482 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:37:32.746486 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:37:32.747844 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:37:32.749424 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:37:32.749430 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:37:32.749432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:37:32.749607 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:37:32.749784 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:37:32.749784 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:37:32.749785 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:37:32.749837 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:37:32.750074 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:37:32.750079 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:37:32.750127 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 02:37:32.750198 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 02:37:32.750310 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260928 02:37:32.750830 16397 tablet_service.cc:1514] Processing DeleteTablet for tablet 29c2dfaf9edf4ac3b9477caae7b7624f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:37:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37236
I20260928 02:37:32.751394 16490 tablet_replica.cc:330] stopping tablet replica
I20260928 02:37:32.751487 16490 raft_consensus.cc:2229] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:37:32.751552 16490 raft_consensus.cc:2258] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:37:32.751840 16490 ts_tablet_manager.cc:1905] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:37:32.753746 16490 ts_tablet_manager.cc:1918] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:37:32.753805 16490 log.cc:1196] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/wal/wals/29c2dfaf9edf4ac3b9477caae7b7624f
I20260928 02:37:32.753978 16490 ts_tablet_manager.cc:1939] T 29c2dfaf9edf4ac3b9477caae7b7624f P d5f2781c1daf49dab4f6ee269fb52c16: Deleting consensus metadata
I20260928 02:37:32.754391 16277 catalog_manager.cc:4841] TS d5f2781c1daf49dab4f6ee269fb52c16 (127.0.98.1:45601): tablet 29c2dfaf9edf4ac3b9477caae7b7624f (table test-table [id=4e362469b30e40d790d0960b0a58bc33]) successfully deleted
W20260928 02:37:32.812521 16485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:37:32.812703 16485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:37:32.812723 16485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:37:32.813808 16485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:37:32.813843 16485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:37:32.813869 16485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:37:32.815196 16485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42391
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43899
--tserver_master_addrs=127.0.98.62:35419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:37:32.815372 16485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:37:32.815541 16485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:37:32.820034 16495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:37:32.820066 16496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:37:32.820072 16485 server_base.cc:1034] running on GCE node
W20260928 02:37:32.820058 16498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:37:32.820390 16485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:37:32.820601 16485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:37:32.821727 16485 hybrid_clock.cc:648] HybridClock initialized: now 1790563052821715 us; error 37 us; skew 500 ppm
Sep 28 02:37:32 dist-test-slave-fz47 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:37:32.823961 16485 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 02:37:32.824342 16485 webserver.cc:460] Webserver started at http://127.0.98.2:43899/ using document root <none> and password file <none>
I20260928 02:37:32.824505 16485 fs_manager.cc:359] Metadata directory not provided
I20260928 02:37:32.824551 16485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:37:32.825759 16485 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:37:32.826228 16505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:32.826382 16485 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:32.826457 16485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "3d4d5723230048c292ecfa086a24fad7"
format_stamp: "Formatted at 2026-09-28 02:37:12 on dist-test-slave-fz47"
server_key: "9cf975d98b8fe32d936fc3755746fa9e"
server_key_iv: "8b92e6ae33b0600db87f9badd28b87f2"
server_key_version: "encryptionkey@0"
I20260928 02:37:32.826745 16485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:37:32.840034 16485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:37:32.841840 16485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:37:32.841943 16485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:37:32.842114 16485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:37:32.842340 16485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:37:32.842367 16485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:32.842386 16485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:37:32.842402 16485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:32.847697 16485 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42391
I20260928 02:37:32.847739 16618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42391 every 8 connection(s)
I20260928 02:37:32.848044 16485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 02:37:32 dist-test-slave-fz47 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:37:32.855320 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 16485
I20260928 02:37:32.855562 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45175
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35419
--webserver_port=43385
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:37:32.856328 16484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:32.848681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51357 (local address 127.0.98.62:35419)
0928 02:37:32.848735 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 02:37:32.848737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:37:32.848957 (+ 220us) server_negotiation.cc:407] Connection header received
0928 02:37:32.849485 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:37:32.849487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:37:32.849502 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:37:32.849548 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:37:32.850436 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:32.850919 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:32.851856 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:32.852052 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:32.852106 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:32.853997 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:37:32.853999 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:37:32.853999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:37:32.854006 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:37:32.855071 (+ 1065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:37:32.855445 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:37:32.855447 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:37:32.855447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:37:32.855497 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:37:32.855737 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:37:32.855738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:37:32.855738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:37:32.855854 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:37:32.855886 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0928 02:37:32.856149 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 02:37:32.856239 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 02:37:32.856513 16621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:32.848690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.2:51357)
0928 02:37:32.848847 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:32.848881 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:37:32.849396 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:37:32.849627 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:37:32.849631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:37:32.850157 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:37:32.850347 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:32.850352 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:32.850997 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:32.850998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:32.851780 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:32.851781 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:32.852118 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:32.852119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:32.852241 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:32.852655 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:37:32.852662 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:37:32.853913 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:37:32.855140 (+ 1227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:37:32.855146 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:37:32.855168 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:37:32.855380 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:37:32.855550 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:37:32.855551 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:37:32.855551 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:37:32.855662 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:37:32.855920 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:37:32.855924 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:37:32.856031 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 02:37:32.856111 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:37:32.856253 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
I20260928 02:37:32.857066 16619 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35419
I20260928 02:37:32.857154 16619 heartbeater.cc:464] Registering TS with master...
I20260928 02:37:32.857352 16619 heartbeater.cc:511] Master 127.0.98.62:35419 requested a full tablet report, sending...
I20260928 02:37:32.857712 16293 ts_manager.cc:194] Registered new tserver with Master: 3d4d5723230048c292ecfa086a24fad7 (127.0.98.2:42391)
I20260928 02:37:32.858166 16293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51357
W20260928 02:37:32.933820 16622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:37:32.934011 16622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:37:32.934033 16622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:37:32.935107 16622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:37:32.935150 16622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:37:32.935174 16622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:37:32.936549 16622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43385
--tserver_master_addrs=127.0.98.62:35419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:37:32.936735 16622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:37:32.936904 16622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:37:32.941709 16622 server_base.cc:1034] running on GCE node
W20260928 02:37:32.941695 16630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:37:32.941757 16628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:37:32.941694 16627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:37:32.942116 16622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:37:32.942325 16622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:37:32.943454 16622 hybrid_clock.cc:648] HybridClock initialized: now 1790563052943440 us; error 33 us; skew 500 ppm
Sep 28 02:37:32 dist-test-slave-fz47 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:37:32.945844 16622 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 02:37:32.946225 16622 webserver.cc:460] Webserver started at http://127.0.98.3:43385/ using document root <none> and password file <none>
I20260928 02:37:32.946445 16622 fs_manager.cc:359] Metadata directory not provided
I20260928 02:37:32.946501 16622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:37:32.947698 16622 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:37:32.948190 16637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:32.948344 16622 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:32.948422 16622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "8c164ae459ed4011874be6e0ea85ecad"
format_stamp: "Formatted at 2026-09-28 02:37:12 on dist-test-slave-fz47"
server_key: "60535341bc26bb63150fa3e3ce09e069"
server_key_iv: "5e63be34f400282bc75e900160eb40b0"
server_key_version: "encryptionkey@0"
I20260928 02:37:32.948721 16622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:37:32.962965 16622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:37:32.964586 16622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:37:32.964689 16622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:37:32.964859 16622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:37:32.965097 16622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:37:32.965127 16622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:32.965152 16622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:37:32.965167 16622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:32.970335 16622 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45175
I20260928 02:37:32.970409 16750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45175 every 8 connection(s)
I20260928 02:37:32.970659 16622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 02:37:32 dist-test-slave-fz47 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:37:32.978775 16484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:32.971245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42013 (local address 127.0.98.62:35419)
0928 02:37:32.971290 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 02:37:32.971292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:37:32.971487 (+ 195us) server_negotiation.cc:407] Connection header received
0928 02:37:32.971975 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:37:32.971977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:37:32.971990 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:37:32.972039 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:37:32.972909 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:32.973445 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:32.974382 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:32.974580 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:32.974630 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:32.976444 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:37:32.976446 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:37:32.976446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:37:32.976454 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:37:32.977525 (+ 1071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:37:32.977865 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:37:32.977867 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:37:32.977867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:37:32.977919 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:37:32.978115 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:37:32.978116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:37:32.978116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:37:32.978224 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:37:32.978276 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 02:37:32.978579 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 02:37:32.978666 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:37:32.978873 16753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:32.971229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.3:42013)
0928 02:37:32.971376 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:32.971410 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:37:32.971881 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:37:32.972103 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:37:32.972107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:37:32.972644 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:37:32.972820 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:32.972825 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:32.973528 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:32.973529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:32.974293 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:32.974295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:32.974641 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:32.974642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:32.974759 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:32.975136 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:37:32.975143 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:37:32.976358 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:37:32.977591 (+ 1233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:37:32.977597 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:37:32.977598 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:37:32.977795 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:37:32.977970 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:37:32.977971 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:37:32.977971 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:37:32.978063 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:37:32.978274 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:37:32.978280 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:37:32.978417 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 02:37:32.978513 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 02:37:32.978670 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260928 02:37:32.979319 16751 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35419
I20260928 02:37:32.979406 16751 heartbeater.cc:464] Registering TS with master...
I20260928 02:37:32.979569 16751 heartbeater.cc:511] Master 127.0.98.62:35419 requested a full tablet report, sending...
I20260928 02:37:32.979902 16293 ts_manager.cc:194] Registered new tserver with Master: 8c164ae459ed4011874be6e0ea85ecad (127.0.98.3:45175)
I20260928 02:37:32.980244 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 16622
I20260928 02:37:32.980348 16293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42013
I20260928 02:37:32.986050 16484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:32.980595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43010 (local address 127.0.98.62:35419)
0928 02:37:32.980637 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 02:37:32.980638 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:37:32.980855 (+ 217us) server_negotiation.cc:407] Connection header received
0928 02:37:32.980909 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:37:32.980910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:37:32.980918 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:37:32.980959 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:37:32.981558 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:32.981999 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:32.982762 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:32.982935 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:32.982972 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:32.984016 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:37:32.984017 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:37:32.984017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:37:32.984021 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:37:32.985093 (+ 1072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:37:32.985469 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:37:32.985469 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:37:32.985470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:37:32.985513 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:37:32.985653 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:37:32.985653 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:37:32.985654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:37:32.985741 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:37:32.985857 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 02:37:32.985940 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 02:37:32.985992 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 02:37:32.986765 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:37:32.988952 16484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:32.987045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43020 (local address 127.0.98.62:35419)
0928 02:37:32.987096 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 02:37:32.987098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:37:32.987152 (+ 54us) server_negotiation.cc:407] Connection header received
0928 02:37:32.987204 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:37:32.987204 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:37:32.987232 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:37:32.987281 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:37:32.987354 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:32.987775 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:32.988424 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:32.988591 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:32.988629 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:32.988719 (+ 90us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:37:32.988803 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:37:32.988866 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:37:32.988882 (+ 16us) server_negotiation.cc:299] Negotiation successful
0928 02:37:32.988894 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 02:37:32.989224 16293 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:37:32.990787 16293 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:37:32.992962 16397 tablet_service.cc:1467] Processing CreateTablet for tablet e6e0b04328a74633b6cffea306278c28 (DEFAULT_TABLE table=test-table [id=78fa72100121442a8269b56154014928]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:37:32.993157 16397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6e0b04328a74633b6cffea306278c28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:37:32.994658 16373 tablet_bootstrap.cc:492] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16: Bootstrap starting.
I20260928 02:37:32.994952 16373 tablet_bootstrap.cc:654] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16: Neither blocks nor log segments found. Creating new log.
I20260928 02:37:32.995467 16373 tablet_bootstrap.cc:492] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16: No bootstrap required, opened a new log
I20260928 02:37:32.995517 16373 ts_tablet_manager.cc:1397] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 02:37:32.995839 16373 raft_consensus.cc:348] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d4d5723230048c292ecfa086a24fad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } } peers { permanent_uuid: "8c164ae459ed4011874be6e0ea85ecad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45175 } } peers { permanent_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45601 } }
I20260928 02:37:32.995916 16373 raft_consensus.cc:374] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:37:32.995929 16488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:32.992558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42391 (local address 127.0.0.1:56192)
0928 02:37:32.992606 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:32.992616 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:37:32.992651 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:37:32.993068 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:37:32.993069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:37:32.993071 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:37:32.993111 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:32.993113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:32.994020 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:32.994021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:32.994913 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:32.994915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:32.995679 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:32.995680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:32.995747 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:32.995766 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 02:37:32.995822 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 02:37:32.995864 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 02:37:32.995946 16758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:32.992713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56192 (local address 127.0.98.2:42391)
0928 02:37:32.992918 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 02:37:32.992922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:37:32.992929 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:37:32.992961 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:37:32.992963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:37:32.992974 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:37:32.993042 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:37:32.993166 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:32.993948 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:32.994984 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:32.995613 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:32.995678 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:32.995737 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 02:37:32.995836 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 02:37:32.995871 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":108,"threads_started":1}
I20260928 02:37:32.995952 16373 raft_consensus.cc:729] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5f2781c1daf49dab4f6ee269fb52c16, State: Initialized, Role: FOLLOWER
I20260928 02:37:32.995996 16373 consensus_queue.cc:260] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d4d5723230048c292ecfa086a24fad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } } peers { permanent_uuid: "8c164ae459ed4011874be6e0ea85ecad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45175 } } peers { permanent_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45601 } }
I20260928 02:37:32.996083 16373 ts_tablet_manager.cc:1428] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 02:37:32.996685 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:32.992558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45175 (local address 127.0.0.1:44060)
0928 02:37:32.992795 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:32.992804 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:37:32.992862 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:37:32.993136 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:37:32.993137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:37:32.993139 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:37:32.993182 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:32.993184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:32.994406 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:32.994407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:32.995464 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:32.995467 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:32.996470 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:32.996471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:32.996536 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:32.996539 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:37:32.996582 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 02:37:32.996619 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":129,"thread_start_us":57,"threads_started":1}
I20260928 02:37:32.996711 16757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:32.992679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44060 (local address 127.0.98.3:45175)
0928 02:37:32.992836 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 02:37:32.992841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:37:32.992852 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:37:32.992955 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:37:32.992958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:37:32.992974 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:37:32.993026 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:37:32.993277 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:32.994254 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:32.995562 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:32.996267 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:32.996309 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:32.996366 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 02:37:32.996608 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 02:37:32.996641 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
I20260928 02:37:32.996716 16553 tablet_service.cc:1467] Processing CreateTablet for tablet e6e0b04328a74633b6cffea306278c28 (DEFAULT_TABLE table=test-table [id=78fa72100121442a8269b56154014928]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:37:32.997020 16553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6e0b04328a74633b6cffea306278c28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:37:32.997814 16685 tablet_service.cc:1467] Processing CreateTablet for tablet e6e0b04328a74633b6cffea306278c28 (DEFAULT_TABLE table=test-table [id=78fa72100121442a8269b56154014928]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:37:32.998067 16685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6e0b04328a74633b6cffea306278c28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:37:32.999634 16760 tablet_bootstrap.cc:492] T e6e0b04328a74633b6cffea306278c28 P 3d4d5723230048c292ecfa086a24fad7: Bootstrap starting.
I20260928 02:37:33.000315 16760 tablet_bootstrap.cc:654] T e6e0b04328a74633b6cffea306278c28 P 3d4d5723230048c292ecfa086a24fad7: Neither blocks nor log segments found. Creating new log.
I20260928 02:37:33.000553 16761 tablet_bootstrap.cc:492] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad: Bootstrap starting.
I20260928 02:37:33.000818 16760 log.cc:824] T e6e0b04328a74633b6cffea306278c28 P 3d4d5723230048c292ecfa086a24fad7: Log is configured to *not* fsync() on all Append() calls
I20260928 02:37:33.001063 16761 tablet_bootstrap.cc:654] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad: Neither blocks nor log segments found. Creating new log.
I20260928 02:37:33.001483 16760 tablet_bootstrap.cc:492] T e6e0b04328a74633b6cffea306278c28 P 3d4d5723230048c292ecfa086a24fad7: No bootstrap required, opened a new log
I20260928 02:37:33.001537 16760 ts_tablet_manager.cc:1397] T e6e0b04328a74633b6cffea306278c28 P 3d4d5723230048c292ecfa086a24fad7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:37:33.001587 16761 log.cc:824] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad: Log is configured to *not* fsync() on all Append() calls
I20260928 02:37:33.002221 16761 tablet_bootstrap.cc:492] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad: No bootstrap required, opened a new log
I20260928 02:37:33.002281 16761 ts_tablet_manager.cc:1397] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:37:33.003163 16760 raft_consensus.cc:348] T e6e0b04328a74633b6cffea306278c28 P 3d4d5723230048c292ecfa086a24fad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d4d5723230048c292ecfa086a24fad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } } peers { permanent_uuid: "8c164ae459ed4011874be6e0ea85ecad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45175 } } peers { permanent_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45601 } }
I20260928 02:37:33.003293 16760 raft_consensus.cc:374] T e6e0b04328a74633b6cffea306278c28 P 3d4d5723230048c292ecfa086a24fad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:37:33.003320 16760 raft_consensus.cc:729] T e6e0b04328a74633b6cffea306278c28 P 3d4d5723230048c292ecfa086a24fad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d4d5723230048c292ecfa086a24fad7, State: Initialized, Role: FOLLOWER
I20260928 02:37:33.003430 16760 consensus_queue.cc:260] T e6e0b04328a74633b6cffea306278c28 P 3d4d5723230048c292ecfa086a24fad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d4d5723230048c292ecfa086a24fad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } } peers { permanent_uuid: "8c164ae459ed4011874be6e0ea85ecad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45175 } } peers { permanent_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45601 } }
I20260928 02:37:33.003683 16760 ts_tablet_manager.cc:1428] T e6e0b04328a74633b6cffea306278c28 P 3d4d5723230048c292ecfa086a24fad7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 02:37:33.003706 16619 heartbeater.cc:503] Master 127.0.98.62:35419 was elected leader, sending a full tablet report...
I20260928 02:37:33.003767 16761 raft_consensus.cc:348] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d4d5723230048c292ecfa086a24fad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } } peers { permanent_uuid: "8c164ae459ed4011874be6e0ea85ecad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45175 } } peers { permanent_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45601 } }
I20260928 02:37:33.003913 16761 raft_consensus.cc:374] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:37:33.003944 16761 raft_consensus.cc:729] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c164ae459ed4011874be6e0ea85ecad, State: Initialized, Role: FOLLOWER
I20260928 02:37:33.004065 16761 consensus_queue.cc:260] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d4d5723230048c292ecfa086a24fad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } } peers { permanent_uuid: "8c164ae459ed4011874be6e0ea85ecad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45175 } } peers { permanent_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45601 } }
I20260928 02:37:33.004303 16761 ts_tablet_manager.cc:1428] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:37:33.004334 16751 heartbeater.cc:503] Master 127.0.98.62:35419 was elected leader, sending a full tablet report...
I20260928 02:37:33.016307 16765 raft_consensus.cc:482] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:37:33.016413 16765 raft_consensus.cc:504] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d4d5723230048c292ecfa086a24fad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } } peers { permanent_uuid: "8c164ae459ed4011874be6e0ea85ecad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45175 } } peers { permanent_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45601 } }
I20260928 02:37:33.016707 16765 leader_election.cc:288] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d4d5723230048c292ecfa086a24fad7 (127.0.98.2:42391), d5f2781c1daf49dab4f6ee269fb52c16 (127.0.98.1:45601)
I20260928 02:37:33.019232 16753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:33.016827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42391 (local address 127.0.98.3:58909)
0928 02:37:33.016879 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:33.016889 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:37:33.016927 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:37:33.017081 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:37:33.017082 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:37:33.017084 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:37:33.017144 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:33.017145 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:33.017776 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:33.017776 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:33.018488 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:33.018490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:33.019005 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:33.019006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:33.019076 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:33.019079 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:37:33.019121 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 02:37:33.019159 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 02:37:33.019230 16758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:33.016924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58909 (local address 127.0.98.2:42391)
0928 02:37:33.016965 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 02:37:33.016967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:37:33.016974 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:37:33.016999 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:37:33.016999 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:37:33.017009 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:37:33.017060 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:37:33.017200 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:33.017708 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:33.018568 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:33.018939 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:33.018997 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:33.019007 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 02:37:33.019133 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 02:37:33.019168 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:37:33.019479 16489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:33.016985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59017 (local address 127.0.98.1:45601)
0928 02:37:33.017035 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 02:37:33.017037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:37:33.017060 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:37:33.017116 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:37:33.017117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:37:33.017140 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:37:33.017196 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:37:33.017384 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:33.017871 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:33.018744 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:33.019132 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:33.019189 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:33.019261 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:37:33.019363 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 02:37:33.019410 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 02:37:33.019598 16573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6e0b04328a74633b6cffea306278c28" candidate_uuid: "8c164ae459ed4011874be6e0ea85ecad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d4d5723230048c292ecfa086a24fad7" is_pre_election: true
I20260928 02:37:33.019670 16766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:33.016857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45601 (local address 127.0.98.3:59017)
0928 02:37:33.017015 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:33.017023 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:37:33.017069 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:37:33.017203 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:37:33.017203 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:37:33.017205 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:37:33.017311 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:33.017312 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:33.017936 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:33.017937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:33.018665 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:33.018667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:33.019200 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:33.019201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:33.019265 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:33.019270 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:37:33.019562 (+ 292us) client_negotiation.cc:240] Negotiation successful
0928 02:37:33.019604 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":94,"threads_started":1}
I20260928 02:37:33.019737 16573 raft_consensus.cc:2454] T e6e0b04328a74633b6cffea306278c28 P 3d4d5723230048c292ecfa086a24fad7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c164ae459ed4011874be6e0ea85ecad in term 0.
I20260928 02:37:33.019958 16639 leader_election.cc:302] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d4d5723230048c292ecfa086a24fad7, 8c164ae459ed4011874be6e0ea85ecad; no voters:
I20260928 02:37:33.019965 16429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6e0b04328a74633b6cffea306278c28" candidate_uuid: "8c164ae459ed4011874be6e0ea85ecad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" is_pre_election: true
I20260928 02:37:33.020089 16429 raft_consensus.cc:2454] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c164ae459ed4011874be6e0ea85ecad in term 0.
I20260928 02:37:33.020073 16765 raft_consensus.cc:2781] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:37:33.020140 16765 raft_consensus.cc:482] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:37:33.020172 16765 raft_consensus.cc:3037] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:37:33.020824 16765 raft_consensus.cc:504] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d4d5723230048c292ecfa086a24fad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } } peers { permanent_uuid: "8c164ae459ed4011874be6e0ea85ecad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45175 } } peers { permanent_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45601 } }
I20260928 02:37:33.020951 16765 leader_election.cc:288] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad [CANDIDATE]: Term 1 election: Requested vote from peers 3d4d5723230048c292ecfa086a24fad7 (127.0.98.2:42391), d5f2781c1daf49dab4f6ee269fb52c16 (127.0.98.1:45601)
I20260928 02:37:33.021091 16573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6e0b04328a74633b6cffea306278c28" candidate_uuid: "8c164ae459ed4011874be6e0ea85ecad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d4d5723230048c292ecfa086a24fad7"
I20260928 02:37:33.021106 16429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6e0b04328a74633b6cffea306278c28" candidate_uuid: "8c164ae459ed4011874be6e0ea85ecad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5f2781c1daf49dab4f6ee269fb52c16"
I20260928 02:37:33.021169 16573 raft_consensus.cc:3037] T e6e0b04328a74633b6cffea306278c28 P 3d4d5723230048c292ecfa086a24fad7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:37:33.021188 16429 raft_consensus.cc:3037] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:37:33.022027 16429 raft_consensus.cc:2454] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c164ae459ed4011874be6e0ea85ecad in term 1.
I20260928 02:37:33.022027 16573 raft_consensus.cc:2454] T e6e0b04328a74633b6cffea306278c28 P 3d4d5723230048c292ecfa086a24fad7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c164ae459ed4011874be6e0ea85ecad in term 1.
I20260928 02:37:33.022163 16639 leader_election.cc:302] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d4d5723230048c292ecfa086a24fad7, 8c164ae459ed4011874be6e0ea85ecad; no voters:
I20260928 02:37:33.022241 16765 raft_consensus.cc:2781] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:37:33.022396 16765 raft_consensus.cc:686] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad [term 1 LEADER]: Becoming Leader. State: Replica: 8c164ae459ed4011874be6e0ea85ecad, State: Running, Role: LEADER
I20260928 02:37:33.022478 16765 consensus_queue.cc:237] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d4d5723230048c292ecfa086a24fad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } } peers { permanent_uuid: "8c164ae459ed4011874be6e0ea85ecad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45175 } } peers { permanent_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45601 } }
I20260928 02:37:33.023185 16292 catalog_manager.cc:5495] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c164ae459ed4011874be6e0ea85ecad (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8c164ae459ed4011874be6e0ea85ecad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d4d5723230048c292ecfa086a24fad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c164ae459ed4011874be6e0ea85ecad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45601 } health_report { overall_health: UNKNOWN } } }
I20260928 02:37:33.032791 16757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:33.031017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44076 (local address 127.0.98.3:45175)
0928 02:37:33.031057 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:37:33.031059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:37:33.031067 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:37:33.031100 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:37:33.031100 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:37:33.031115 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:37:33.031166 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:37:33.031245 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:33.031675 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:33.032291 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:33.032461 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:33.032502 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:33.032595 (+ 93us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:37:33.032634 (+ 39us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 02:37:33.032730 (+ 96us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44076: token expired
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 02:37:33.032835 16757 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44076: token expired
I20260928 02:37:33.038053 16484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:33.033103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43030 (local address 127.0.98.62:35419)
0928 02:37:33.033141 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 02:37:33.033143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:37:33.033165 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:37:33.033219 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:37:33.033220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:37:33.033229 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:37:33.033291 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:37:33.033732 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:33.034178 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:33.034796 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:33.034982 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:33.035032 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:33.036041 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:37:33.036043 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:37:33.036043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:37:33.036050 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:37:33.037074 (+ 1024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:37:33.037406 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:37:33.037407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:37:33.037407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:37:33.037460 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:37:33.037616 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:37:33.037616 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:37:33.037617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:37:33.037716 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:37:33.037797 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:37:33.037899 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 02:37:33.037970 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:37:33.040457 16757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:33.038811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44078 (local address 127.0.98.3:45175)
0928 02:37:33.038847 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 02:37:33.038848 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:37:33.038856 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:37:33.038877 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:37:33.038877 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:37:33.038886 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:37:33.038930 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:37:33.038994 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:33.039401 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:33.039977 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:33.040146 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:33.040185 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:33.040259 (+ 74us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:37:33.040331 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:37:33.040362 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0928 02:37:33.040398 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 02:37:33.040412 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 02:37:33.041975 16573 raft_consensus.cc:1261] T e6e0b04328a74633b6cffea306278c28 P 3d4d5723230048c292ecfa086a24fad7 [term 1 FOLLOWER]: Refusing update from remote peer 8c164ae459ed4011874be6e0ea85ecad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:37:33.041989 16429 raft_consensus.cc:1261] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [term 1 FOLLOWER]: Refusing update from remote peer 8c164ae459ed4011874be6e0ea85ecad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:37:33.042253 16767 consensus_queue.cc:1035] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d4d5723230048c292ecfa086a24fad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:37:33.042364 16765 consensus_queue.cc:1035] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:37:33.044440 16771 mvcc.cc:205] Tried to move back new op lower bound from 7334146265257885696 to 7334146265180729344. Current Snapshot: MvccSnapshot[applied={T|T < 7334146265257885696}]
I20260928 02:37:33.044804 16773 mvcc.cc:205] Tried to move back new op lower bound from 7334146265257885696 to 7334146265180729344. Current Snapshot: MvccSnapshot[applied={T|T < 7334146265257885696}]
I20260928 02:37:33.046211 16292 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:37:33.046303 16292 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43030:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:37:33.046953 16292 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5: Sending DeleteTablet for 3 replicas of tablet e6e0b04328a74633b6cffea306278c28
I20260928 02:37:33.047271 16553 tablet_service.cc:1514] Processing DeleteTablet for tablet e6e0b04328a74633b6cffea306278c28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:37:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56192
I20260928 02:37:33.047266 16685 tablet_service.cc:1514] Processing DeleteTablet for tablet e6e0b04328a74633b6cffea306278c28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:37:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44060
I20260928 02:37:33.047490 16397 tablet_service.cc:1514] Processing DeleteTablet for tablet e6e0b04328a74633b6cffea306278c28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:37:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37236
I20260928 02:37:33.047590 16490 tablet_replica.cc:330] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16: stopping tablet replica
I20260928 02:37:33.047593 16780 tablet_replica.cc:330] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad: stopping tablet replica
I20260928 02:37:33.047642 16490 raft_consensus.cc:2229] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:37:33.047678 16780 raft_consensus.cc:2229] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:37:33.047698 16490 raft_consensus.cc:2258] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:37:33.047780 16780 raft_consensus.cc:2258] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:37:33.047868 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 16331
I20260928 02:37:33.047933 16490 ts_tablet_manager.cc:1905] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:37:33.048197 16780 ts_tablet_manager.cc:1905] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:37:33.049751 16779 tablet_replica.cc:330] T e6e0b04328a74633b6cffea306278c28 P 3d4d5723230048c292ecfa086a24fad7: stopping tablet replica
I20260928 02:37:33.049851 16779 raft_consensus.cc:2229] T e6e0b04328a74633b6cffea306278c28 P 3d4d5723230048c292ecfa086a24fad7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:37:33.049944 16779 raft_consensus.cc:2258] T e6e0b04328a74633b6cffea306278c28 P 3d4d5723230048c292ecfa086a24fad7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:37:33.050350 16779 ts_tablet_manager.cc:1905] T e6e0b04328a74633b6cffea306278c28 P 3d4d5723230048c292ecfa086a24fad7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:37:33.052006 16779 ts_tablet_manager.cc:1918] T e6e0b04328a74633b6cffea306278c28 P 3d4d5723230048c292ecfa086a24fad7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:37:33.052073 16779 log.cc:1196] T e6e0b04328a74633b6cffea306278c28 P 3d4d5723230048c292ecfa086a24fad7: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/wal/wals/e6e0b04328a74633b6cffea306278c28
I20260928 02:37:33.052379 16779 ts_tablet_manager.cc:1939] T e6e0b04328a74633b6cffea306278c28 P 3d4d5723230048c292ecfa086a24fad7: Deleting consensus metadata
I20260928 02:37:33.052899 16278 catalog_manager.cc:4841] TS 3d4d5723230048c292ecfa086a24fad7 (127.0.98.2:42391): tablet e6e0b04328a74633b6cffea306278c28 (table test-table [id=78fa72100121442a8269b56154014928]) successfully deleted
W20260928 02:37:33.053300 16277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45601 (error 108)
I20260928 02:37:33.053637 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 16485
I20260928 02:37:33.053785 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:33.053676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45601 (local address 127.0.0.1:37248)
0928 02:37:33.053723 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:33.053756 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 02:37:33.053908 16277 catalog_manager.cc:4584] TS d5f2781c1daf49dab4f6ee269fb52c16 (127.0.98.1:45601): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e6e0b04328a74633b6cffea306278c28: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45601: connect: Connection refused (error 111)
I20260928 02:37:33.057375 16780 ts_tablet_manager.cc:1918] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:37:33.057451 16780 log.cc:1196] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/wal/wals/e6e0b04328a74633b6cffea306278c28
I20260928 02:37:33.057699 16780 ts_tablet_manager.cc:1939] T e6e0b04328a74633b6cffea306278c28 P 8c164ae459ed4011874be6e0ea85ecad: Deleting consensus metadata
I20260928 02:37:33.058064 16280 catalog_manager.cc:4841] TS 8c164ae459ed4011874be6e0ea85ecad (127.0.98.3:45175): tablet e6e0b04328a74633b6cffea306278c28 (table test-table [id=78fa72100121442a8269b56154014928]) successfully deleted
I20260928 02:37:33.059187 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 16622
I20260928 02:37:33.063387 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 16260
I20260928 02:37:53.068259 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35419
--webserver_interface=127.0.98.62
--webserver_port=43591
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:37:53.145367 16782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:37:53.145562 16782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:37:53.145591 16782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:37:53.146696 16782 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 02:37:53.146731 16782 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 02:37:53.146747 16782 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 02:37:53.146761 16782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:37:53.146777 16782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:37:53.146801 16782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:37:53.146824 16782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:37:53.146840 16782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:37:53.148319 16782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:37:53.148550 16782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:37:53.148741 16782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:37:53.153522 16788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:37:53.153576 16787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:37:53.153527 16790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:37:53.153594 16782 server_base.cc:1034] running on GCE node
I20260928 02:37:53.153949 16782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:37:53.154208 16782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:37:53.155349 16782 hybrid_clock.cc:648] HybridClock initialized: now 1790563073155344 us; error 26 us; skew 500 ppm
Sep 28 02:37:53 dist-test-slave-fz47 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:37:53.157613 16782 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:37:53.158038 16782 webserver.cc:460] Webserver started at http://127.0.98.62:43591/ using document root <none> and password file <none>
I20260928 02:37:53.158236 16782 fs_manager.cc:359] Metadata directory not provided
I20260928 02:37:53.158283 16782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:37:53.159507 16782 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:37:53.160039 16797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:53.160194 16782 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:53.160269 16782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "f4550d46bc52494b98423de6c092eda5"
format_stamp: "Formatted at 2026-09-28 02:37:11 on dist-test-slave-fz47"
server_key: "dfa125bd31d63d79cb520a3cb79e5d1b"
server_key_iv: "333db4266a747246188820117fa19827"
server_key_version: "encryptionkey@0"
I20260928 02:37:53.160620 16782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:37:53.170079 16782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:37:53.171595 16782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:37:53.171715 16782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:37:53.175266 16782 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35419
I20260928 02:37:53.175294 16849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35419 every 8 connection(s)
I20260928 02:37:53.175621 16782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:37:53.176928 16850 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:37:53.177578 16850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5: Bootstrap starting.
I20260928 02:37:53.182647 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 16782
I20260928 02:37:53.182982 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45601
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35419
--webserver_port=43739
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:37:53.185974 16850 log.cc:824] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:37:53.187942 16850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:37:53.188167 16850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5: Bootstrap complete.
I20260928 02:37:53.190188 16850 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4550d46bc52494b98423de6c092eda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35419 } }
I20260928 02:37:53.190407 16850 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4550d46bc52494b98423de6c092eda5, State: Initialized, Role: FOLLOWER
I20260928 02:37:53.190541 16850 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4550d46bc52494b98423de6c092eda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35419 } }
I20260928 02:37:53.190594 16850 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:37:53.190632 16850 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:37:53.190675 16850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [term 2 FOLLOWER]: Advancing to term 3
I20260928 02:37:53.191587 16850 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4550d46bc52494b98423de6c092eda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35419 } }
I20260928 02:37:53.191669 16850 leader_election.cc:302] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4550d46bc52494b98423de6c092eda5; no voters:
I20260928 02:37:53.191812 16850 leader_election.cc:288] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 02:37:53.191857 16854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 02:37:53.192004 16854 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [term 3 LEADER]: Becoming Leader. State: Replica: f4550d46bc52494b98423de6c092eda5, State: Running, Role: LEADER
I20260928 02:37:53.192061 16850 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:37:53.192122 16854 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4550d46bc52494b98423de6c092eda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35419 } }
I20260928 02:37:53.192404 16855 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f4550d46bc52494b98423de6c092eda5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4550d46bc52494b98423de6c092eda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35419 } } }
I20260928 02:37:53.192435 16854 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4550d46bc52494b98423de6c092eda5. Latest consensus state: current_term: 3 leader_uuid: "f4550d46bc52494b98423de6c092eda5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4550d46bc52494b98423de6c092eda5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35419 } } }
I20260928 02:37:53.192546 16855 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [sys.catalog]: This master's current role is: LEADER
I20260928 02:37:53.192574 16854 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5 [sys.catalog]: This master's current role is: LEADER
I20260928 02:37:53.192773 16860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:37:53.193645 16860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:37:53.193815 16860 catalog_manager.cc:1259] Loaded cluster ID: e72c588e73ad413da116559d40605921
I20260928 02:37:53.193996 16860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:37:53.194803 16860 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:37:53.194954 16860 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5: Loaded TSK: 0
I20260928 02:37:53.194990 16860 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f4550d46bc52494b98423de6c092eda5: Loaded TSK: 1
F20260928 02:37:53.195194 16860 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb1908179ad google::LogMessage::Fail() at ??:0
@ 0x7fb19081a88a google::LogMessage::SendToLog() at ??:0
@ 0x7fb1908174de google::LogMessage::Flush() at ??:0
@ 0x7fb190818a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb192f84104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fb192f81d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fb1987711cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fb19877148e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fb198771594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fb198737c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fb198741ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb1926e50d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb1926da954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb19353e6db start_thread at ??:0
@ 0x7fb18f31561f clone at ??:0
W20260928 02:37:53.274829 16852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:37:53.275019 16852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:37:53.275049 16852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:37:53.276129 16852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:37:53.276167 16852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:37:53.276209 16852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:37:53.277670 16852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43739
--tserver_master_addrs=127.0.98.62:35419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:37:53.277897 16852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:37:53.278092 16852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:37:53.282691 16894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:37:53.282701 16891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:37:53.282935 16852 server_base.cc:1034] running on GCE node
W20260928 02:37:53.282965 16892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:37:53.283159 16852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:37:53.283366 16852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:37:53.284513 16852 hybrid_clock.cc:648] HybridClock initialized: now 1790563073284498 us; error 29 us; skew 500 ppm
Sep 28 02:37:53 dist-test-slave-fz47 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:37:53.286998 16852 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 02:37:53.287436 16852 webserver.cc:460] Webserver started at http://127.0.98.1:43739/ using document root <none> and password file <none>
I20260928 02:37:53.287635 16852 fs_manager.cc:359] Metadata directory not provided
I20260928 02:37:53.287681 16852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:37:53.288965 16852 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:37:53.289695 16901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:53.289868 16852 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:53.289976 16852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "d5f2781c1daf49dab4f6ee269fb52c16"
format_stamp: "Formatted at 2026-09-28 02:37:11 on dist-test-slave-fz47"
server_key: "272e473f1948e79243f152b942d38319"
server_key_iv: "3f18022403f17ff9c653e7d8c5e3787e"
server_key_version: "encryptionkey@0"
I20260928 02:37:53.290371 16852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:37:53.305542 16852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:37:53.307519 16852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:37:53.307646 16852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:37:53.307838 16852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:37:53.308244 16908 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 02:37:53.310146 16852 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:37:53.310187 16852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 02:37:53.310218 16852 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:37:53.310812 16852 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:37:53.310842 16852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 02:37:53.310940 16908 tablet_bootstrap.cc:492] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16: Bootstrap starting.
I20260928 02:37:53.312657 16908 log.cc:824] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16: Log is configured to *not* fsync() on all Append() calls
I20260928 02:37:53.315475 16908 tablet_bootstrap.cc:492] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:37:53.315740 16908 tablet_bootstrap.cc:492] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16: Bootstrap complete.
I20260928 02:37:53.315896 16908 ts_tablet_manager.cc:1397] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260928 02:37:53.317252 16852 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45601
I20260928 02:37:53.317327 17016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45601 every 8 connection(s)
I20260928 02:37:53.317591 16852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
I20260928 02:37:53.317889 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 16852
I20260928 02:37:53.317700 16908 raft_consensus.cc:348] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d4d5723230048c292ecfa086a24fad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } } peers { permanent_uuid: "8c164ae459ed4011874be6e0ea85ecad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45175 } } peers { permanent_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45601 } }
I20260928 02:37:53.318070 16908 raft_consensus.cc:729] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5f2781c1daf49dab4f6ee269fb52c16, State: Initialized, Role: FOLLOWER
I20260928 02:37:53.318145 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42391
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35419
--webserver_port=43899
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:37:53.318229 16908 consensus_queue.cc:260] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d4d5723230048c292ecfa086a24fad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } } peers { permanent_uuid: "8c164ae459ed4011874be6e0ea85ecad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45175 } } peers { permanent_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45601 } }
I20260928 02:37:53.318501 16908 ts_tablet_manager.cc:1428] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
W20260928 02:37:53.380311 17021 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:37:53.319087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.1:57229)
0928 02:37:53.319246 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:53.319295 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:37:53.319455 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:37:53.380111 (+ 60656us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
W20260928 02:37:53.380584 17017 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35419 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35419: Client connection negotiation failed: client connection to 127.0.98.62:35419: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:37:53.380856 17021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:53.380764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.1:44563)
0928 02:37:53.380804 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:53.380835 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 02:37:53.381137 17021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:53.380999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.1:39469)
0928 02:37:53.381100 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:53.381118 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":72,"threads_started":1}
W20260928 02:37:53.381206 17017 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:37:53.400568 17020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:37:53.400763 17020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:37:53.400785 17020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:37:53.401906 17020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:37:53.401952 17020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:37:53.401975 17020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:37:53.403345 17020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42391
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43899
--tserver_master_addrs=127.0.98.62:35419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:37:53.403527 17020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:37:53.403697 17020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:37:53.408520 17020 server_base.cc:1034] running on GCE node
W20260928 02:37:53.408499 17030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:37:53.408556 17028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:37:53.408682 17027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:37:53.408892 17020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:37:53.409108 17020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:37:53.410238 17020 hybrid_clock.cc:648] HybridClock initialized: now 1790563073410218 us; error 34 us; skew 500 ppm
Sep 28 02:37:53 dist-test-slave-fz47 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:37:53.412426 17020 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 02:37:53.412801 17020 webserver.cc:460] Webserver started at http://127.0.98.2:43899/ using document root <none> and password file <none>
I20260928 02:37:53.413007 17020 fs_manager.cc:359] Metadata directory not provided
I20260928 02:37:53.413054 17020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:37:53.414336 17020 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:37:53.414891 17037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:53.415055 17020 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:53.415143 17020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "3d4d5723230048c292ecfa086a24fad7"
format_stamp: "Formatted at 2026-09-28 02:37:12 on dist-test-slave-fz47"
server_key: "9cf975d98b8fe32d936fc3755746fa9e"
server_key_iv: "8b92e6ae33b0600db87f9badd28b87f2"
server_key_version: "encryptionkey@0"
I20260928 02:37:53.415473 17020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:37:53.427424 17020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:37:53.428997 17020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:37:53.429123 17020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:37:53.429340 17020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:37:53.429612 17020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:37:53.429644 17020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:53.429670 17020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:37:53.429694 17020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:53.435072 17020 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42391
I20260928 02:37:53.435125 17150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42391 every 8 connection(s)
I20260928 02:37:53.435410 17020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
I20260928 02:37:53.436286 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:53.435962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.2:47253)
0928 02:37:53.436087 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:53.436154 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":61,"threads_started":1}
W20260928 02:37:53.436610 17151 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35419 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35419: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
I20260928 02:37:53.436836 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:53.436762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.2:36009)
0928 02:37:53.436796 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:53.436821 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 02:37:53.437042 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:53.436987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.2:47459)
0928 02:37:53.437014 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:53.437031 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260928 02:37:53.437116 17151 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:37:53.445418 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 17020
I20260928 02:37:53.445629 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45175
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35419
--webserver_port=43385
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:37:53.522801 17154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:37:53.522984 17154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:37:53.523005 17154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:37:53.524086 17154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:37:53.524128 17154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:37:53.524153 17154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:37:53.525588 17154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43385
--tserver_master_addrs=127.0.98.62:35419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:37:53.525787 17154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:37:53.525985 17154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:37:53.530386 17159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:37:53.530435 17160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:37:53.530543 17154 server_base.cc:1034] running on GCE node
W20260928 02:37:53.530581 17162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:37:53.530771 17154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:37:53.530963 17154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:37:53.532114 17154 hybrid_clock.cc:648] HybridClock initialized: now 1790563073532101 us; error 25 us; skew 500 ppm
Sep 28 02:37:53 dist-test-slave-fz47 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:37:53.534437 17154 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 02:37:53.534857 17154 webserver.cc:460] Webserver started at http://127.0.98.3:43385/ using document root <none> and password file <none>
I20260928 02:37:53.535053 17154 fs_manager.cc:359] Metadata directory not provided
I20260928 02:37:53.535095 17154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:37:53.536284 17154 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:37:53.536756 17169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:53.536902 17154 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:53.536979 17154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "8c164ae459ed4011874be6e0ea85ecad"
format_stamp: "Formatted at 2026-09-28 02:37:12 on dist-test-slave-fz47"
server_key: "60535341bc26bb63150fa3e3ce09e069"
server_key_iv: "5e63be34f400282bc75e900160eb40b0"
server_key_version: "encryptionkey@0"
I20260928 02:37:53.537292 17154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:37:53.545585 17154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:37:53.547124 17154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:37:53.547223 17154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:37:53.547374 17154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:37:53.547587 17154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:37:53.547614 17154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:53.547633 17154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:37:53.547648 17154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:37:53.552883 17154 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45175
I20260928 02:37:53.552925 17282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45175 every 8 connection(s)
I20260928 02:37:53.553201 17154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
I20260928 02:37:53.554126 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:53.553805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.3:37665)
0928 02:37:53.553937 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:53.554001 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":65,"threads_started":1}
W20260928 02:37:53.554476 17283 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35419 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35419: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
I20260928 02:37:53.554699 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:53.554625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.3:46561)
0928 02:37:53.554659 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:53.554684 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 02:37:53.554908 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:53.554849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.3:55689)
0928 02:37:53.554878 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:53.554896 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260928 02:37:53.554983 17283 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:37:53.559737 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 17154
I20260928 02:37:54.381721 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:54.381500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.1:40337)
0928 02:37:54.381635 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:54.381672 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260928 02:37:54.437604 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:54.437425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.2:43071)
0928 02:37:54.437527 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:54.437558 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":56,"threads_started":1}
I20260928 02:37:54.555474 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:54.555283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.3:49655)
0928 02:37:54.555391 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:54.555426 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":57,"threads_started":1}
I20260928 02:37:54.890024 17290 raft_consensus.cc:482] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:37:54.890143 17290 raft_consensus.cc:504] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d4d5723230048c292ecfa086a24fad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } } peers { permanent_uuid: "8c164ae459ed4011874be6e0ea85ecad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45175 } } peers { permanent_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45601 } }
I20260928 02:37:54.890483 17290 leader_election.cc:288] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d4d5723230048c292ecfa086a24fad7 (127.0.98.2:42391), 8c164ae459ed4011874be6e0ea85ecad (127.0.98.3:45175)
Sep 28 02:37:54 dist-test-slave-fz47 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 28 02:37:54 dist-test-slave-fz47 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260928 02:37:54.902731 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:54.890745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50081 (local address 127.0.98.2:42391)
0928 02:37:54.890924 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 02:37:54.890937 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:37:54.890977 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:37:54.891534 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:37:54.891537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:37:54.891596 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:37:54.891746 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:37:54.892727 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:54.894153 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:54.895686 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:54.896317 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:54.896435 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:54.899206 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:37:54.899211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:37:54.899211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:37:54.899225 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:37:54.900903 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:37:54.901375 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:37:54.901379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:37:54.901379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:37:54.901492 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:37:54.901759 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:37:54.901760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:37:54.901760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:37:54.902068 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:37:54.902152 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 02:37:54.902518 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 02:37:54.902633 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":64,"threads_started":1}
I20260928 02:37:54.902731 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:54.890751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41775 (local address 127.0.98.3:45175)
0928 02:37:54.890929 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 02:37:54.890937 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:37:54.890977 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:37:54.891534 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:37:54.891537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:37:54.891596 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:37:54.891746 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:37:54.892880 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:54.894210 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:54.895685 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:37:54.896317 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:37:54.896436 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:54.899472 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:37:54.899475 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:37:54.899476 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:37:54.899486 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:37:54.900919 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:37:54.901446 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:37:54.901448 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:37:54.901449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:37:54.901542 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:37:54.901781 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:37:54.901781 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:37:54.901782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:37:54.902068 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:37:54.902152 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 02:37:54.902478 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 02:37:54.902615 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":57,"threads_started":1}
I20260928 02:37:54.902812 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:54.890606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45175 (local address 127.0.98.1:41775)
0928 02:37:54.890736 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:54.890746 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:37:54.890835 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:37:54.891815 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:37:54.891818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:37:54.892293 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:37:54.892794 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:54.892796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:54.894293 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:54.894294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:54.895609 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:54.895612 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:54.896412 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:54.896414 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:54.896563 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:54.897186 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:37:54.897197 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:37:54.899382 (+ 2185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:37:54.901087 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:37:54.901088 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:37:54.901088 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:37:54.901372 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:37:54.901603 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:37:54.901604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:37:54.901604 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:37:54.901698 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:37:54.902137 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:37:54.902143 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:37:54.902301 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 02:37:54.902417 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 02:37:54.902653 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":141,"thread_start_us":47,"threads_started":1}
I20260928 02:37:54.903028 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:54.890621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42391 (local address 127.0.98.1:50081)
0928 02:37:54.890829 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:54.890837 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:37:54.890875 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:37:54.891815 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:37:54.891818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:37:54.892293 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:37:54.892614 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:54.892623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:54.894250 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:54.894251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:54.895609 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:37:54.895612 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:37:54.896414 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:37:54.896415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:37:54.896563 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:37:54.897186 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:37:54.897197 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:37:54.899070 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:37:54.900991 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:37:54.900998 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:37:54.901001 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:37:54.901295 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:37:54.901557 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:37:54.901558 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:37:54.901558 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:37:54.901698 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:37:54.902137 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:37:54.902143 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:37:54.902301 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 02:37:54.902417 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 02:37:54.902592 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":93,"thread_start_us":51,"threads_started":1}
W20260928 02:37:54.903678 16905 leader_election.cc:341] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8c164ae459ed4011874be6e0ea85ecad (127.0.98.3:45175): Not found: Tablet not found: e6e0b04328a74633b6cffea306278c28
W20260928 02:37:54.904767 16903 leader_election.cc:341] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3d4d5723230048c292ecfa086a24fad7 (127.0.98.2:42391): Not found: Tablet not found: e6e0b04328a74633b6cffea306278c28
I20260928 02:37:54.904817 16903 leader_election.cc:302] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5f2781c1daf49dab4f6ee269fb52c16; no voters: 3d4d5723230048c292ecfa086a24fad7, 8c164ae459ed4011874be6e0ea85ecad
I20260928 02:37:54.904914 17290 raft_consensus.cc:2726] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:37:55.382263 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:55.382124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.1:59901)
0928 02:37:55.382185 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:55.382220 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 02:37:55.438166 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:55.438000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.2:43879)
0928 02:37:55.438107 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:55.438134 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":57,"threads_started":1}
I20260928 02:37:55.556008 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:55.555825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.3:59623)
0928 02:37:55.555936 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:55.555973 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":53,"threads_started":1}
I20260928 02:37:56.382879 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:56.382672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.1:41853)
0928 02:37:56.382794 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:56.382828 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":64,"threads_started":1}
I20260928 02:37:56.438773 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:56.438571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.2:46973)
0928 02:37:56.438700 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:56.438735 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
I20260928 02:37:56.556571 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:56.556371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.3:55221)
0928 02:37:56.556501 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:56.556533 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
I20260928 02:37:57.097692 17300 raft_consensus.cc:482] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:37:57.097786 17300 raft_consensus.cc:504] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d4d5723230048c292ecfa086a24fad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } } peers { permanent_uuid: "8c164ae459ed4011874be6e0ea85ecad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45175 } } peers { permanent_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45601 } }
I20260928 02:37:57.097945 17300 leader_election.cc:288] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d4d5723230048c292ecfa086a24fad7 (127.0.98.2:42391), 8c164ae459ed4011874be6e0ea85ecad (127.0.98.3:45175)
W20260928 02:37:57.098376 16905 leader_election.cc:341] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8c164ae459ed4011874be6e0ea85ecad (127.0.98.3:45175): Not found: Tablet not found: e6e0b04328a74633b6cffea306278c28
W20260928 02:37:57.098444 16903 leader_election.cc:341] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3d4d5723230048c292ecfa086a24fad7 (127.0.98.2:42391): Not found: Tablet not found: e6e0b04328a74633b6cffea306278c28
I20260928 02:37:57.098544 16903 leader_election.cc:302] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5f2781c1daf49dab4f6ee269fb52c16; no voters: 3d4d5723230048c292ecfa086a24fad7, 8c164ae459ed4011874be6e0ea85ecad
I20260928 02:37:57.098636 17300 raft_consensus.cc:2726] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:37:57.383464 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:57.383269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.1:32997)
0928 02:37:57.383384 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:57.383417 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
I20260928 02:37:57.439348 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:57.439142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.2:38971)
0928 02:37:57.439266 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:57.439300 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
I20260928 02:37:57.557214 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:57.556991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.3:44263)
0928 02:37:57.557125 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:57.557160 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":62,"threads_started":1}
I20260928 02:37:58.384110 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:58.383871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.1:52399)
0928 02:37:58.384012 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:58.384065 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260928 02:37:58.439917 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:58.439703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.2:41503)
0928 02:37:58.439834 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:58.439878 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":58,"threads_started":1}
I20260928 02:37:58.557782 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:58.557578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.3:35523)
0928 02:37:58.557710 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:58.557744 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
I20260928 02:37:59.034853 17307 raft_consensus.cc:482] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:37:59.034942 17307 raft_consensus.cc:504] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d4d5723230048c292ecfa086a24fad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } } peers { permanent_uuid: "8c164ae459ed4011874be6e0ea85ecad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45175 } } peers { permanent_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45601 } }
I20260928 02:37:59.035110 17307 leader_election.cc:288] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d4d5723230048c292ecfa086a24fad7 (127.0.98.2:42391), 8c164ae459ed4011874be6e0ea85ecad (127.0.98.3:45175)
W20260928 02:37:59.035421 16903 leader_election.cc:341] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3d4d5723230048c292ecfa086a24fad7 (127.0.98.2:42391): Not found: Tablet not found: e6e0b04328a74633b6cffea306278c28
W20260928 02:37:59.035506 16905 leader_election.cc:341] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8c164ae459ed4011874be6e0ea85ecad (127.0.98.3:45175): Not found: Tablet not found: e6e0b04328a74633b6cffea306278c28
I20260928 02:37:59.035605 16905 leader_election.cc:302] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5f2781c1daf49dab4f6ee269fb52c16; no voters: 3d4d5723230048c292ecfa086a24fad7, 8c164ae459ed4011874be6e0ea85ecad
I20260928 02:37:59.035691 17307 raft_consensus.cc:2726] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:37:59.384745 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:59.384513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.1:60677)
0928 02:37:59.384646 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:59.384689 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260928 02:37:59.440479 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:59.440289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.2:41969)
0928 02:37:59.440402 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:59.440439 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":59,"threads_started":1}
I20260928 02:37:59.558396 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:37:59.558190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.3:59051)
0928 02:37:59.558306 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 02:37:59.558353 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":57,"threads_started":1}
I20260928 02:38:00.385588 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:00.385382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.1:38981)
0928 02:38:00.385505 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:00.385541 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260928 02:38:00.441066 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:00.440877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.2:59905)
0928 02:38:00.440987 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:00.441025 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":56,"threads_started":1}
I20260928 02:38:00.559017 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:00.558787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.3:36809)
0928 02:38:00.558943 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:00.558979 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":78,"threads_started":1}
I20260928 02:38:00.826928 17314 raft_consensus.cc:482] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:38:00.827020 17314 raft_consensus.cc:504] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d4d5723230048c292ecfa086a24fad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } } peers { permanent_uuid: "8c164ae459ed4011874be6e0ea85ecad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45175 } } peers { permanent_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45601 } }
I20260928 02:38:00.827179 17314 leader_election.cc:288] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d4d5723230048c292ecfa086a24fad7 (127.0.98.2:42391), 8c164ae459ed4011874be6e0ea85ecad (127.0.98.3:45175)
W20260928 02:38:00.827508 16905 leader_election.cc:341] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8c164ae459ed4011874be6e0ea85ecad (127.0.98.3:45175): Not found: Tablet not found: e6e0b04328a74633b6cffea306278c28
W20260928 02:38:00.827589 16903 leader_election.cc:341] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3d4d5723230048c292ecfa086a24fad7 (127.0.98.2:42391): Not found: Tablet not found: e6e0b04328a74633b6cffea306278c28
I20260928 02:38:00.827629 16903 leader_election.cc:302] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5f2781c1daf49dab4f6ee269fb52c16; no voters: 3d4d5723230048c292ecfa086a24fad7, 8c164ae459ed4011874be6e0ea85ecad
I20260928 02:38:00.827703 17314 raft_consensus.cc:2726] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:38:01.386198 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:01.385974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.1:60235)
0928 02:38:01.386116 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:01.386153 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":65,"threads_started":1}
I20260928 02:38:01.441651 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:01.441452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.2:54059)
0928 02:38:01.441573 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:01.441611 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
I20260928 02:38:01.559597 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:01.559409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.3:35695)
0928 02:38:01.559518 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:01.559553 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":58,"threads_started":1}
I20260928 02:38:02.386883 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:02.386648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.1:55781)
0928 02:38:02.386801 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:02.386838 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":87,"threads_started":1}
I20260928 02:38:02.442245 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:02.442060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.2:47577)
0928 02:38:02.442170 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:02.442205 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":55,"threads_started":1}
I20260928 02:38:02.560271 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:02.560045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.3:39669)
0928 02:38:02.560181 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:02.560221 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
I20260928 02:38:03.387607 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:03.387355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.1:53699)
0928 02:38:03.387505 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:03.387548 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":87,"threads_started":1}
I20260928 02:38:03.442869 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:03.442668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.2:41919)
0928 02:38:03.442787 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:03.442826 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260928 02:38:03.560850 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:03.560639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.3:34697)
0928 02:38:03.560772 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:03.560805 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20260928 02:38:04.388285 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:04.388085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.1:57467)
0928 02:38:04.388201 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:04.388239 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260928 02:38:04.443459 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:04.443270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.2:35911)
0928 02:38:04.443377 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:04.443415 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":58,"threads_started":1}
I20260928 02:38:04.561514 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:04.561311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.3:34173)
0928 02:38:04.561437 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:04.561473 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
I20260928 02:38:05.388826 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:05.388642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.1:44993)
0928 02:38:05.388747 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:05.388778 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":60,"threads_started":1}
I20260928 02:38:05.444111 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:05.443883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.2:40601)
0928 02:38:05.443994 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:05.444045 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":58,"threads_started":1}
I20260928 02:38:05.562122 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:05.561905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.3:44665)
0928 02:38:05.562034 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:05.562076 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":57,"threads_started":1}
I20260928 02:38:06.389451 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:06.389204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.1:47113)
0928 02:38:06.389369 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:06.389403 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":115,"threads_started":1}
I20260928 02:38:06.444672 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:06.444475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.2:60355)
0928 02:38:06.444596 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:06.444631 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
I20260928 02:38:06.562748 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:06.562551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.3:56519)
0928 02:38:06.562661 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:06.562703 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":59,"threads_started":1}
I20260928 02:38:07.040513 17333 raft_consensus.cc:482] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:38:07.040611 17333 raft_consensus.cc:504] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d4d5723230048c292ecfa086a24fad7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42391 } } peers { permanent_uuid: "8c164ae459ed4011874be6e0ea85ecad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45175 } } peers { permanent_uuid: "d5f2781c1daf49dab4f6ee269fb52c16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45601 } }
I20260928 02:38:07.040776 17333 leader_election.cc:288] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d4d5723230048c292ecfa086a24fad7 (127.0.98.2:42391), 8c164ae459ed4011874be6e0ea85ecad (127.0.98.3:45175)
W20260928 02:38:07.041117 16903 leader_election.cc:341] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3d4d5723230048c292ecfa086a24fad7 (127.0.98.2:42391): Not found: Tablet not found: e6e0b04328a74633b6cffea306278c28
W20260928 02:38:07.041205 16905 leader_election.cc:341] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8c164ae459ed4011874be6e0ea85ecad (127.0.98.3:45175): Not found: Tablet not found: e6e0b04328a74633b6cffea306278c28
I20260928 02:38:07.041283 16905 leader_election.cc:302] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5f2781c1daf49dab4f6ee269fb52c16; no voters: 3d4d5723230048c292ecfa086a24fad7, 8c164ae459ed4011874be6e0ea85ecad
I20260928 02:38:07.041352 17333 raft_consensus.cc:2726] T e6e0b04328a74633b6cffea306278c28 P d5f2781c1daf49dab4f6ee269fb52c16 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:38:07.390157 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:07.389945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.1:38659)
0928 02:38:07.390079 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:07.390111 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":65,"threads_started":1}
I20260928 02:38:07.445305 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:07.445043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.2:47995)
0928 02:38:07.445192 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:07.445233 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":78,"threads_started":1}
I20260928 02:38:07.563371 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:07.563171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.3:57773)
0928 02:38:07.563285 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:07.563324 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
I20260928 02:38:08.390854 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:08.390644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.1:38477)
0928 02:38:08.390776 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:08.390810 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260928 02:38:08.445972 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:08.445777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.2:55569)
0928 02:38:08.445887 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:08.445922 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":64,"threads_started":1}
I20260928 02:38:08.562105 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 16852
I20260928 02:38:08.563998 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:08.563790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35419 (local address 127.0.98.3:53793)
0928 02:38:08.563907 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:08.563951 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":48,"threads_started":1}
I20260928 02:38:08.566191 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 17020
I20260928 02:38:08.570535 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 17154
2026-09-28T02:38:08Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58028 ms)
[----------] 30 tests from SecurityITest (225771 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:38:08 dist-test-slave-fz47 krb5kdc[17342](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:38:08 dist-test-slave-fz47 krb5kdc[17342](info): set up 2 sockets
Sep 28 02:38:08 dist-test-slave-fz47 krb5kdc[17342](info): commencing operation
krb5kdc: starting...
W20260928 02:38:09.520668 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:38:09 dist-test-slave-fz47 krb5kdc[17342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:38:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:38:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 02:38:09.628175 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34099
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:09.704861 17355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:09.705060 17355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:09.705087 17355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:09.706246 17355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:09.706288 17355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:09.706307 17355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:09.706323 17355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:09.706337 17355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:09.707793 17355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:09.708020 17355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:09.708213 17355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:09.712891 17360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:09.712903 17361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:09.712906 17363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:09.712973 17355 server_base.cc:1034] running on GCE node
I20260928 02:38:09.713338 17355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:09.713591 17355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:09.714754 17355 hybrid_clock.cc:648] HybridClock initialized: now 1790563089714740 us; error 25 us; skew 500 ppm
Sep 28 02:38:09 dist-test-slave-fz47 krb5kdc[17342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:09.716988 17355 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:38:09.717447 17355 webserver.cc:460] Webserver started at http://127.0.98.62:35815/ using document root <none> and password file <none>
I20260928 02:38:09.717664 17355 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:09.717711 17355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:09.717830 17355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:09.718815 17355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "a4dbd81c7f114425a0de25212a16630b"
format_stamp: "Formatted at 2026-09-28 02:38:09 on dist-test-slave-fz47"
server_key: "facb277ee8caa99325d0ad29f0f5ab7d"
server_key_iv: "b29595e328cc5b22cba84c69d6c68f77"
server_key_version: "encryptionkey@0"
I20260928 02:38:09.719120 17355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "a4dbd81c7f114425a0de25212a16630b"
format_stamp: "Formatted at 2026-09-28 02:38:09 on dist-test-slave-fz47"
server_key: "facb277ee8caa99325d0ad29f0f5ab7d"
server_key_iv: "b29595e328cc5b22cba84c69d6c68f77"
server_key_version: "encryptionkey@0"
I20260928 02:38:09.720296 17355 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:38:09.720894 17370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:09.721050 17355 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:38:09.721125 17355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "a4dbd81c7f114425a0de25212a16630b"
format_stamp: "Formatted at 2026-09-28 02:38:09 on dist-test-slave-fz47"
server_key: "facb277ee8caa99325d0ad29f0f5ab7d"
server_key_iv: "b29595e328cc5b22cba84c69d6c68f77"
server_key_version: "encryptionkey@0"
I20260928 02:38:09.721189 17355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:09.730266 17355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:09.731812 17355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:09.731928 17355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:09.735455 17355 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34099
I20260928 02:38:09.735484 17422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34099 every 8 connection(s)
I20260928 02:38:09.735764 17355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:38:09.736495 17423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:09.738639 17423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4dbd81c7f114425a0de25212a16630b: Bootstrap starting.
I20260928 02:38:09.739131 17423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4dbd81c7f114425a0de25212a16630b: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:09.739347 17423 log.cc:824] T 00000000000000000000000000000000 P a4dbd81c7f114425a0de25212a16630b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:09.739923 17423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4dbd81c7f114425a0de25212a16630b: No bootstrap required, opened a new log
I20260928 02:38:09.741446 17423 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4dbd81c7f114425a0de25212a16630b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4dbd81c7f114425a0de25212a16630b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34099 } }
I20260928 02:38:09.741570 17423 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4dbd81c7f114425a0de25212a16630b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:09.741590 17423 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4dbd81c7f114425a0de25212a16630b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4dbd81c7f114425a0de25212a16630b, State: Initialized, Role: FOLLOWER
I20260928 02:38:09.741685 17423 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4dbd81c7f114425a0de25212a16630b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4dbd81c7f114425a0de25212a16630b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34099 } }
I20260928 02:38:09.741730 17423 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4dbd81c7f114425a0de25212a16630b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:38:09.741748 17423 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4dbd81c7f114425a0de25212a16630b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:38:09.741776 17423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4dbd81c7f114425a0de25212a16630b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:09.742174 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 17355
I20260928 02:38:09.742276 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:38:09.742292 17423 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4dbd81c7f114425a0de25212a16630b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4dbd81c7f114425a0de25212a16630b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34099 } }
I20260928 02:38:09.742388 17423 leader_election.cc:302] T 00000000000000000000000000000000 P a4dbd81c7f114425a0de25212a16630b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4dbd81c7f114425a0de25212a16630b; no voters:
I20260928 02:38:09.742475 392 external_mini_cluster.cc:1428] Setting key d0e10d54c2e083b90ffa8703dadf8157
I20260928 02:38:09.742540 17423 leader_election.cc:288] T 00000000000000000000000000000000 P a4dbd81c7f114425a0de25212a16630b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:38:09.742571 17426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4dbd81c7f114425a0de25212a16630b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:09.742703 17426 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4dbd81c7f114425a0de25212a16630b [term 1 LEADER]: Becoming Leader. State: Replica: a4dbd81c7f114425a0de25212a16630b, State: Running, Role: LEADER
I20260928 02:38:09.742769 17423 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4dbd81c7f114425a0de25212a16630b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:38:09.742877 17426 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4dbd81c7f114425a0de25212a16630b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4dbd81c7f114425a0de25212a16630b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34099 } }
I20260928 02:38:09.743796 17427 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4dbd81c7f114425a0de25212a16630b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4dbd81c7f114425a0de25212a16630b. Latest consensus state: current_term: 1 leader_uuid: "a4dbd81c7f114425a0de25212a16630b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4dbd81c7f114425a0de25212a16630b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34099 } } }
I20260928 02:38:09.743883 17427 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4dbd81c7f114425a0de25212a16630b [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:09.743959 17428 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4dbd81c7f114425a0de25212a16630b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4dbd81c7f114425a0de25212a16630b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4dbd81c7f114425a0de25212a16630b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34099 } } }
I20260928 02:38:09.744030 17428 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4dbd81c7f114425a0de25212a16630b [sys.catalog]: This master's current role is: LEADER
W20260928 02:38:09.744617 17444 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a4dbd81c7f114425a0de25212a16630b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:38:09.744668 17444 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:38:09.744709 17441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:38:09.744853 17441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:38:09.746192 17441 catalog_manager.cc:1347] Generated new cluster ID: a71d6b9cbddb4c60a413f8548b4d1cb2
I20260928 02:38:09.746238 17441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:38:09 dist-test-slave-fz47 krb5kdc[17342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:38:09.752341 17441 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:38:09.753074 17441 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:38:09.753319 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:09.742931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34964 (local address 127.0.98.62:34099)
0928 02:38:09.743123 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:09.743133 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:09.743167 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:38:09.743281 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:09.743288 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:09.743341 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:09.744276 (+ 935us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:09.744378 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:09.745523 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:09.746533 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:09.747105 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:09.747204 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:09.749778 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:09.749782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:09.749783 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:09.749795 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:09.751636 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:09.752116 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:09.752119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:09.752120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:09.752217 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:09.752420 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:09.752421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:09.752422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:09.752633 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:09.752780 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:09.752873 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 02:38:09.753067 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":151,"thread_start_us":61,"threads_started":1}
I20260928 02:38:09.756839 17441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4dbd81c7f114425a0de25212a16630b: Generated new TSK 0
I20260928 02:38:09.756999 17441 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 02:38:09.784180 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34099
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:09.870298 17449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:09.870501 17449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:09.870523 17449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:09.871670 17449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:09.871712 17449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:09.871737 17449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:38:09.873186 17449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:09.873435 17449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:09.873639 17449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:09.879189 17454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:09.879305 17449 server_base.cc:1034] running on GCE node
W20260928 02:38:09.879168 17455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:09.879171 17457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:09.879729 17449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:09.879964 17449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:09.881111 17449 hybrid_clock.cc:648] HybridClock initialized: now 1790563089881100 us; error 25 us; skew 500 ppm
Sep 28 02:38:09 dist-test-slave-fz47 krb5kdc[17342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:09.883605 17449 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 02:38:09.884038 17449 webserver.cc:460] Webserver started at http://127.0.98.1:39397/ using document root <none> and password file <none>
I20260928 02:38:09.884234 17449 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:09.884279 17449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:09.884406 17449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:09.885464 17449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-0/data/instance:
uuid: "31c2682bd9d547fbb2821eff519c3e6e"
format_stamp: "Formatted at 2026-09-28 02:38:09 on dist-test-slave-fz47"
server_key: "08b526ed5dac191dc8b9e6870b8ef631"
server_key_iv: "968d5dc9e0e3e3c5dd86e1013473cdda"
server_key_version: "encryptionkey@0"
I20260928 02:38:09.885776 17449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-0/wal/instance:
uuid: "31c2682bd9d547fbb2821eff519c3e6e"
format_stamp: "Formatted at 2026-09-28 02:38:09 on dist-test-slave-fz47"
server_key: "08b526ed5dac191dc8b9e6870b8ef631"
server_key_iv: "968d5dc9e0e3e3c5dd86e1013473cdda"
server_key_version: "encryptionkey@0"
I20260928 02:38:09.886941 17449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:38:09.887583 17464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:09.887715 17449 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:09.887794 17449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "31c2682bd9d547fbb2821eff519c3e6e"
format_stamp: "Formatted at 2026-09-28 02:38:09 on dist-test-slave-fz47"
server_key: "08b526ed5dac191dc8b9e6870b8ef631"
server_key_iv: "968d5dc9e0e3e3c5dd86e1013473cdda"
server_key_version: "encryptionkey@0"
I20260928 02:38:09.887852 17449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:09.914333 17449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:09.916038 17449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:09.916163 17449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:09.916376 17449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:38:09.916658 17449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:38:09.916690 17449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:09.916718 17449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:38:09.916734 17449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:09.922246 17449 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35945
I20260928 02:38:09.922297 17577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35945 every 8 connection(s)
I20260928 02:38:09.922592 17449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 02:38:09 dist-test-slave-fz47 krb5kdc[17342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:38:09.929065 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 17449
I20260928 02:38:09.929171 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-0/wal/instance
I20260928 02:38:09.929378 392 external_mini_cluster.cc:1428] Setting key 229f0cc777863337e293ccad21a4dc1b
I20260928 02:38:09.931063 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:09.923204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51599 (local address 127.0.98.62:34099)
0928 02:38:09.923278 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:09.923280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:09.923426 (+ 146us) server_negotiation.cc:407] Connection header received
0928 02:38:09.924370 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:09.924371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:09.924380 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:09.924429 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:09.925019 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:09.925548 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:09.926513 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:09.926698 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:09.926738 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:09.928546 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:09.928547 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:09.928548 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:09.928554 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:09.929905 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:09.930244 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:09.930246 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:09.930246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:09.930298 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:09.930494 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:09.930495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:09.930496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:09.930597 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:09.930650 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:09.930895 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 02:38:09.930973 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:38:09.931209 17580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:09.923182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34099 (local address 127.0.98.1:51599)
0928 02:38:09.923316 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:09.923351 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:09.924266 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:09.924484 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:09.924488 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:09.924765 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:09.924950 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:09.924956 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:09.925605 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:09.925606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:09.926453 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:09.926456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:09.926762 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:09.926763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:09.926855 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:09.927248 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:09.927255 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:09.928465 (+ 1210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:09.929977 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:09.929983 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:09.929985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:09.930179 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:09.930350 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:09.930351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:09.930351 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:09.930442 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:09.930649 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:09.930655 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:09.930780 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 02:38:09.930865 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 02:38:09.931004 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":69,"threads_started":1}
I20260928 02:38:09.931674 17578 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34099
I20260928 02:38:09.931761 17578 heartbeater.cc:464] Registering TS with master...
I20260928 02:38:09.931917 17578 heartbeater.cc:511] Master 127.0.98.62:34099 requested a full tablet report, sending...
I20260928 02:38:09.932456 17387 ts_manager.cc:194] Registered new tserver with Master: 31c2682bd9d547fbb2821eff519c3e6e (127.0.98.1:35945)
I20260928 02:38:09.933175 17387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51599
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 02:38:09.951804 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34099
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:10.030390 17583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:10.030583 17583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:10.030606 17583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:10.031675 17583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:10.031708 17583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:10.031733 17583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:38:10.033118 17583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:10.033342 17583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:10.033524 17583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:38:10.038126 17583 server_base.cc:1034] running on GCE node
W20260928 02:38:10.038129 17589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:10.038151 17588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:10.038291 17591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:10.038532 17583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:10.038745 17583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:10.039876 17583 hybrid_clock.cc:648] HybridClock initialized: now 1790563090039866 us; error 27 us; skew 500 ppm
Sep 28 02:38:10 dist-test-slave-fz47 krb5kdc[17342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:10.042140 17583 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 02:38:10.042492 17583 webserver.cc:460] Webserver started at http://127.0.98.2:42463/ using document root <none> and password file <none>
I20260928 02:38:10.042660 17583 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:10.042696 17583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:10.042779 17583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:10.043766 17583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-1/data/instance:
uuid: "98d48ece99ec458b83525789b90cb939"
format_stamp: "Formatted at 2026-09-28 02:38:10 on dist-test-slave-fz47"
server_key: "7dbfec3e883e9756f35b658b8f8d0fe9"
server_key_iv: "4d888c10e1174331d107a911fe240ad5"
server_key_version: "encryptionkey@0"
I20260928 02:38:10.044096 17583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-1/wal/instance:
uuid: "98d48ece99ec458b83525789b90cb939"
format_stamp: "Formatted at 2026-09-28 02:38:10 on dist-test-slave-fz47"
server_key: "7dbfec3e883e9756f35b658b8f8d0fe9"
server_key_iv: "4d888c10e1174331d107a911fe240ad5"
server_key_version: "encryptionkey@0"
I20260928 02:38:10.045218 17583 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:38:10.045910 17598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:10.046061 17583 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:10.046142 17583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "98d48ece99ec458b83525789b90cb939"
format_stamp: "Formatted at 2026-09-28 02:38:10 on dist-test-slave-fz47"
server_key: "7dbfec3e883e9756f35b658b8f8d0fe9"
server_key_iv: "4d888c10e1174331d107a911fe240ad5"
server_key_version: "encryptionkey@0"
I20260928 02:38:10.046197 17583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:10.063199 17583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:10.064846 17583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:10.064946 17583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:10.065104 17583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:38:10.065387 17583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:38:10.065419 17583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:10.065446 17583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:38:10.065462 17583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:10.070827 17583 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33811
I20260928 02:38:10.070900 17711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33811 every 8 connection(s)
I20260928 02:38:10.071166 17583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
I20260928 02:38:10.075927 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 17583
I20260928 02:38:10.076042 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-1/wal/instance
I20260928 02:38:10.076259 392 external_mini_cluster.cc:1428] Setting key 5795c614a214bd7cd9714fa1a5a725c3
Sep 28 02:38:10 dist-test-slave-fz47 krb5kdc[17342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:38:10.080672 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:10.071909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38703 (local address 127.0.98.62:34099)
0928 02:38:10.072000 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:10.072002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:10.072159 (+ 157us) server_negotiation.cc:407] Connection header received
0928 02:38:10.073089 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:10.073090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:10.073103 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:10.073155 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:10.073830 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.074320 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.075391 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.075593 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.075649 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:10.077985 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:10.077986 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:10.077986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:10.077993 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:10.079405 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:10.079787 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:10.079788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:10.079788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:10.079847 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:10.080063 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:10.080064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:10.080064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:10.080173 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:10.080237 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:10.080496 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 02:38:10.080584 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:38:10.080840 17714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:10.071848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34099 (local address 127.0.98.2:38703)
0928 02:38:10.072038 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:10.072079 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:10.072982 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:10.073229 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:10.073234 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:10.073541 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:10.073730 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:10.073735 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.074401 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.074402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:10.075305 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:10.075308 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.075659 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.075660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:10.075779 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:10.076275 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:10.076283 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:10.077904 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:10.079472 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:10.079478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:10.079480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:10.079685 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:10.079897 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:10.079898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:10.079898 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:10.080010 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:10.080227 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:10.080233 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:10.080369 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 02:38:10.080476 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 02:38:10.080622 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":83,"threads_started":1}
I20260928 02:38:10.081401 17712 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34099
I20260928 02:38:10.081504 17712 heartbeater.cc:464] Registering TS with master...
I20260928 02:38:10.081696 17712 heartbeater.cc:511] Master 127.0.98.62:34099 requested a full tablet report, sending...
I20260928 02:38:10.082043 17387 ts_manager.cc:194] Registered new tserver with Master: 98d48ece99ec458b83525789b90cb939 (127.0.98.2:33811)
I20260928 02:38:10.082404 17387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38703
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 02:38:10.098033 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34099
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:10.175623 17717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:10.175825 17717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:10.175848 17717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:10.176929 17717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:10.176963 17717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:10.176987 17717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:38:10.178421 17717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:10.178604 17717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:10.178821 17717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:10.183466 17723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:10.183437 17725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:10.183580 17722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:10.183667 17717 server_base.cc:1034] running on GCE node
I20260928 02:38:10.183840 17717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:10.184022 17717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:10.185145 17717 hybrid_clock.cc:648] HybridClock initialized: now 1790563090185133 us; error 26 us; skew 500 ppm
Sep 28 02:38:10 dist-test-slave-fz47 krb5kdc[17342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:10.187515 17717 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 02:38:10.187923 17717 webserver.cc:460] Webserver started at http://127.0.98.3:35999/ using document root <none> and password file <none>
I20260928 02:38:10.188119 17717 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:10.188162 17717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:10.188283 17717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:10.189378 17717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-2/data/instance:
uuid: "7f00fea834bf42a9b822e5750bcf6e23"
format_stamp: "Formatted at 2026-09-28 02:38:10 on dist-test-slave-fz47"
server_key: "bb821995d917d9ec2288bc5a67a63637"
server_key_iv: "f44854611c75686de49490b312b1b8c1"
server_key_version: "encryptionkey@0"
I20260928 02:38:10.189729 17717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7f00fea834bf42a9b822e5750bcf6e23"
format_stamp: "Formatted at 2026-09-28 02:38:10 on dist-test-slave-fz47"
server_key: "bb821995d917d9ec2288bc5a67a63637"
server_key_iv: "f44854611c75686de49490b312b1b8c1"
server_key_version: "encryptionkey@0"
I20260928 02:38:10.191004 17717 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:38:10.191741 17732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:10.191917 17717 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:38:10.192001 17717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "7f00fea834bf42a9b822e5750bcf6e23"
format_stamp: "Formatted at 2026-09-28 02:38:10 on dist-test-slave-fz47"
server_key: "bb821995d917d9ec2288bc5a67a63637"
server_key_iv: "f44854611c75686de49490b312b1b8c1"
server_key_version: "encryptionkey@0"
I20260928 02:38:10.192070 17717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:10.231320 17717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:10.233398 17717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:10.233536 17717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:10.233748 17717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:38:10.234054 17717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:38:10.234092 17717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:10.234122 17717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:38:10.234139 17717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:10.239428 17717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34819
I20260928 02:38:10.239487 17845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34819 every 8 connection(s)
I20260928 02:38:10.239778 17717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
I20260928 02:38:10.242554 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 17717
I20260928 02:38:10.242671 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-2/wal/instance
I20260928 02:38:10.242859 392 external_mini_cluster.cc:1428] Setting key 91a833bff33df3c608a296704d8c1c1d
Sep 28 02:38:10 dist-test-slave-fz47 krb5kdc[17342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:38:10.248503 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:10.240474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60197 (local address 127.0.98.62:34099)
0928 02:38:10.240527 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:10.240529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:10.240694 (+ 165us) server_negotiation.cc:407] Connection header received
0928 02:38:10.241634 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:10.241636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:10.241645 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:10.241698 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:10.242279 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.243005 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.244240 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.244481 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.244539 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:10.246428 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:10.246429 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:10.246430 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:10.246436 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:10.247480 (+ 1044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:10.247800 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:10.247801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:10.247801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:10.247845 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:10.248021 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:10.248021 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:10.248022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:10.248127 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:10.248177 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:10.248382 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 02:38:10.248438 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 02:38:10.248692 17848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:10.240421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34099 (local address 127.0.98.3:60197)
0928 02:38:10.240575 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:10.240616 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:10.241549 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:10.241754 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:10.241758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:10.242005 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:10.242196 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:10.242201 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.243100 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.243101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:10.244177 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:10.244180 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.244549 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.244550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:10.244652 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:10.245048 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:10.245055 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:10.246349 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:10.247541 (+ 1192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:10.247547 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:10.247548 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:10.247746 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:10.247882 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:10.247882 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:10.247882 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:10.247964 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:10.248175 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:10.248177 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:10.248282 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 02:38:10.248362 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:38:10.248472 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260928 02:38:10.249110 17846 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34099
I20260928 02:38:10.249183 17846 heartbeater.cc:464] Registering TS with master...
I20260928 02:38:10.249370 17846 heartbeater.cc:511] Master 127.0.98.62:34099 requested a full tablet report, sending...
I20260928 02:38:10.249722 17387 ts_manager.cc:194] Registered new tserver with Master: 7f00fea834bf42a9b822e5750bcf6e23 (127.0.98.3:34819)
I20260928 02:38:10.250080 17387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60197
I20260928 02:38:10.254212 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:38:10.260593 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:10.255384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34970 (local address 127.0.98.62:34099)
0928 02:38:10.255444 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:10.255446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:10.255521 (+ 75us) server_negotiation.cc:407] Connection header received
0928 02:38:10.255570 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:10.255571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:10.255579 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:10.255613 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:10.256099 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.256547 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.257414 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.257601 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.257680 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:10.258648 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:10.258650 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:10.258650 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:10.258656 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:10.259804 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:10.260048 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:10.260049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:10.260049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:10.260090 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:10.260222 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:10.260222 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:10.260222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:10.260341 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:10.260407 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:10.260475 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 02:38:10.260527 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 02:38:10.261987 17387 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:38:10.263818 17387 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:38:10.269841 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:10.266030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33811 (local address 127.0.0.1:44712)
0928 02:38:10.266169 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:10.266199 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:10.266262 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:10.266617 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:10.266620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:10.266622 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:10.266705 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:10.266707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.267646 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.267647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:10.268716 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:10.268719 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.269537 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.269538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:10.269635 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:10.269648 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 02:38:10.269721 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:38:10.269768 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":43,"threads_started":1}
I20260928 02:38:10.269850 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:10.266281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44712 (local address 127.0.98.2:33811)
0928 02:38:10.266433 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:10.266439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:10.266449 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:38:10.266492 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:10.266494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:10.266510 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:10.266782 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:10.266881 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.267554 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.268781 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.269460 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.269528 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:10.269581 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:10.269738 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 02:38:10.269775 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":70,"threads_started":1}
I20260928 02:38:10.270195 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:10.266036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34819 (local address 127.0.0.1:54082)
0928 02:38:10.266444 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:10.266455 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:10.266512 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:10.266744 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:10.266745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:10.266747 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:10.266799 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:10.266801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.268141 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.268142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:10.269087 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:10.269089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.269965 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.269966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:10.270031 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:10.270035 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:38:10.270085 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 02:38:10.270126 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":87,"threads_started":1}
I20260928 02:38:10.270220 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:10.266257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54082 (local address 127.0.98.3:34819)
0928 02:38:10.266400 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:10.266405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:10.266516 (+ 111us) server_negotiation.cc:407] Connection header received
0928 02:38:10.266574 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:10.266576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:10.266591 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:10.266867 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:10.266936 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.268046 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.269183 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.269863 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.269959 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:10.270017 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:10.270100 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 02:38:10.270140 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
I20260928 02:38:10.270825 17646 tablet_service.cc:1467] Processing CreateTablet for tablet d4fd8cec03894cc9bba813d6298f77b6 (DEFAULT_TABLE table=test-table [id=7a8351ac71134ce5a4b7b0d860d6e24b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:10.271010 17780 tablet_service.cc:1467] Processing CreateTablet for tablet d4fd8cec03894cc9bba813d6298f77b6 (DEFAULT_TABLE table=test-table [id=7a8351ac71134ce5a4b7b0d860d6e24b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:10.271136 17646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4fd8cec03894cc9bba813d6298f77b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:10.271207 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:10.266038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35945 (local address 127.0.0.1:34464)
0928 02:38:10.266569 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:10.266577 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:10.266610 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:10.266834 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:10.266834 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:10.266836 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:10.266863 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:10.266864 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.269163 (+ 2299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.269164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:10.270090 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:10.270093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.270969 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.270970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:10.271042 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:10.271046 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:38:10.271101 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 02:38:10.271137 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":87,"threads_started":1}
I20260928 02:38:10.271224 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:10.266448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34464 (local address 127.0.98.1:35945)
0928 02:38:10.266624 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:10.266628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:10.266637 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:38:10.266672 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:10.266673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:10.266686 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:10.266753 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:10.266931 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.268034 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.270178 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.270883 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.270956 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:10.271013 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:10.271111 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 02:38:10.271152 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":68,"threads_started":1}
I20260928 02:38:10.271332 17780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4fd8cec03894cc9bba813d6298f77b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:10.273717 17863 tablet_bootstrap.cc:492] T d4fd8cec03894cc9bba813d6298f77b6 P 98d48ece99ec458b83525789b90cb939: Bootstrap starting.
I20260928 02:38:10.273837 17864 tablet_bootstrap.cc:492] T d4fd8cec03894cc9bba813d6298f77b6 P 7f00fea834bf42a9b822e5750bcf6e23: Bootstrap starting.
I20260928 02:38:10.273967 17512 tablet_service.cc:1467] Processing CreateTablet for tablet d4fd8cec03894cc9bba813d6298f77b6 (DEFAULT_TABLE table=test-table [id=7a8351ac71134ce5a4b7b0d860d6e24b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:10.274242 17512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4fd8cec03894cc9bba813d6298f77b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:10.274461 17864 tablet_bootstrap.cc:654] T d4fd8cec03894cc9bba813d6298f77b6 P 7f00fea834bf42a9b822e5750bcf6e23: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:10.274461 17863 tablet_bootstrap.cc:654] T d4fd8cec03894cc9bba813d6298f77b6 P 98d48ece99ec458b83525789b90cb939: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:10.275004 17864 log.cc:824] T d4fd8cec03894cc9bba813d6298f77b6 P 7f00fea834bf42a9b822e5750bcf6e23: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:10.275024 17863 log.cc:824] T d4fd8cec03894cc9bba813d6298f77b6 P 98d48ece99ec458b83525789b90cb939: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:10.275709 17864 tablet_bootstrap.cc:492] T d4fd8cec03894cc9bba813d6298f77b6 P 7f00fea834bf42a9b822e5750bcf6e23: No bootstrap required, opened a new log
I20260928 02:38:10.275768 17864 ts_tablet_manager.cc:1397] T d4fd8cec03894cc9bba813d6298f77b6 P 7f00fea834bf42a9b822e5750bcf6e23: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:38:10.275794 17863 tablet_bootstrap.cc:492] T d4fd8cec03894cc9bba813d6298f77b6 P 98d48ece99ec458b83525789b90cb939: No bootstrap required, opened a new log
I20260928 02:38:10.275851 17863 ts_tablet_manager.cc:1397] T d4fd8cec03894cc9bba813d6298f77b6 P 98d48ece99ec458b83525789b90cb939: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:38:10.276960 17867 tablet_bootstrap.cc:492] T d4fd8cec03894cc9bba813d6298f77b6 P 31c2682bd9d547fbb2821eff519c3e6e: Bootstrap starting.
I20260928 02:38:10.277400 17864 raft_consensus.cc:348] T d4fd8cec03894cc9bba813d6298f77b6 P 7f00fea834bf42a9b822e5750bcf6e23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d48ece99ec458b83525789b90cb939" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33811 } } peers { permanent_uuid: "7f00fea834bf42a9b822e5750bcf6e23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34819 } } peers { permanent_uuid: "31c2682bd9d547fbb2821eff519c3e6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35945 } }
I20260928 02:38:10.277563 17864 raft_consensus.cc:374] T d4fd8cec03894cc9bba813d6298f77b6 P 7f00fea834bf42a9b822e5750bcf6e23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:10.277595 17867 tablet_bootstrap.cc:654] T d4fd8cec03894cc9bba813d6298f77b6 P 31c2682bd9d547fbb2821eff519c3e6e: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:10.277599 17864 raft_consensus.cc:729] T d4fd8cec03894cc9bba813d6298f77b6 P 7f00fea834bf42a9b822e5750bcf6e23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f00fea834bf42a9b822e5750bcf6e23, State: Initialized, Role: FOLLOWER
I20260928 02:38:10.277585 17863 raft_consensus.cc:348] T d4fd8cec03894cc9bba813d6298f77b6 P 98d48ece99ec458b83525789b90cb939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d48ece99ec458b83525789b90cb939" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33811 } } peers { permanent_uuid: "7f00fea834bf42a9b822e5750bcf6e23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34819 } } peers { permanent_uuid: "31c2682bd9d547fbb2821eff519c3e6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35945 } }
I20260928 02:38:10.277729 17863 raft_consensus.cc:374] T d4fd8cec03894cc9bba813d6298f77b6 P 98d48ece99ec458b83525789b90cb939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:10.277762 17863 raft_consensus.cc:729] T d4fd8cec03894cc9bba813d6298f77b6 P 98d48ece99ec458b83525789b90cb939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98d48ece99ec458b83525789b90cb939, State: Initialized, Role: FOLLOWER
I20260928 02:38:10.277732 17864 consensus_queue.cc:260] T d4fd8cec03894cc9bba813d6298f77b6 P 7f00fea834bf42a9b822e5750bcf6e23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d48ece99ec458b83525789b90cb939" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33811 } } peers { permanent_uuid: "7f00fea834bf42a9b822e5750bcf6e23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34819 } } peers { permanent_uuid: "31c2682bd9d547fbb2821eff519c3e6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35945 } }
I20260928 02:38:10.277854 17863 consensus_queue.cc:260] T d4fd8cec03894cc9bba813d6298f77b6 P 98d48ece99ec458b83525789b90cb939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d48ece99ec458b83525789b90cb939" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33811 } } peers { permanent_uuid: "7f00fea834bf42a9b822e5750bcf6e23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34819 } } peers { permanent_uuid: "31c2682bd9d547fbb2821eff519c3e6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35945 } }
I20260928 02:38:10.278017 17864 ts_tablet_manager.cc:1428] T d4fd8cec03894cc9bba813d6298f77b6 P 7f00fea834bf42a9b822e5750bcf6e23: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:38:10.278088 17863 ts_tablet_manager.cc:1428] T d4fd8cec03894cc9bba813d6298f77b6 P 98d48ece99ec458b83525789b90cb939: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:38:10.278030 17846 heartbeater.cc:503] Master 127.0.98.62:34099 was elected leader, sending a full tablet report...
I20260928 02:38:10.278112 17867 log.cc:824] T d4fd8cec03894cc9bba813d6298f77b6 P 31c2682bd9d547fbb2821eff519c3e6e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:10.278187 17712 heartbeater.cc:503] Master 127.0.98.62:34099 was elected leader, sending a full tablet report...
I20260928 02:38:10.278981 17867 tablet_bootstrap.cc:492] T d4fd8cec03894cc9bba813d6298f77b6 P 31c2682bd9d547fbb2821eff519c3e6e: No bootstrap required, opened a new log
I20260928 02:38:10.279047 17867 ts_tablet_manager.cc:1397] T d4fd8cec03894cc9bba813d6298f77b6 P 31c2682bd9d547fbb2821eff519c3e6e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 02:38:10.280520 17867 raft_consensus.cc:348] T d4fd8cec03894cc9bba813d6298f77b6 P 31c2682bd9d547fbb2821eff519c3e6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d48ece99ec458b83525789b90cb939" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33811 } } peers { permanent_uuid: "7f00fea834bf42a9b822e5750bcf6e23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34819 } } peers { permanent_uuid: "31c2682bd9d547fbb2821eff519c3e6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35945 } }
I20260928 02:38:10.280651 17867 raft_consensus.cc:374] T d4fd8cec03894cc9bba813d6298f77b6 P 31c2682bd9d547fbb2821eff519c3e6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:10.280682 17867 raft_consensus.cc:729] T d4fd8cec03894cc9bba813d6298f77b6 P 31c2682bd9d547fbb2821eff519c3e6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31c2682bd9d547fbb2821eff519c3e6e, State: Initialized, Role: FOLLOWER
I20260928 02:38:10.280783 17867 consensus_queue.cc:260] T d4fd8cec03894cc9bba813d6298f77b6 P 31c2682bd9d547fbb2821eff519c3e6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d48ece99ec458b83525789b90cb939" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33811 } } peers { permanent_uuid: "7f00fea834bf42a9b822e5750bcf6e23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34819 } } peers { permanent_uuid: "31c2682bd9d547fbb2821eff519c3e6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35945 } }
I20260928 02:38:10.280998 17867 ts_tablet_manager.cc:1428] T d4fd8cec03894cc9bba813d6298f77b6 P 31c2682bd9d547fbb2821eff519c3e6e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 02:38:10.281038 17578 heartbeater.cc:503] Master 127.0.98.62:34099 was elected leader, sending a full tablet report...
W20260928 02:38:10.321712 17713 tablet.cc:2369] T d4fd8cec03894cc9bba813d6298f77b6 P 98d48ece99ec458b83525789b90cb939: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:38:10.423239 17579 tablet.cc:2369] T d4fd8cec03894cc9bba813d6298f77b6 P 31c2682bd9d547fbb2821eff519c3e6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:38:10.490302 17847 tablet.cc:2369] T d4fd8cec03894cc9bba813d6298f77b6 P 7f00fea834bf42a9b822e5750bcf6e23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:38:10.599905 17871 raft_consensus.cc:482] T d4fd8cec03894cc9bba813d6298f77b6 P 31c2682bd9d547fbb2821eff519c3e6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:38:10.600029 17871 raft_consensus.cc:504] T d4fd8cec03894cc9bba813d6298f77b6 P 31c2682bd9d547fbb2821eff519c3e6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d48ece99ec458b83525789b90cb939" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33811 } } peers { permanent_uuid: "7f00fea834bf42a9b822e5750bcf6e23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34819 } } peers { permanent_uuid: "31c2682bd9d547fbb2821eff519c3e6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35945 } }
I20260928 02:38:10.600400 17871 leader_election.cc:288] T d4fd8cec03894cc9bba813d6298f77b6 P 31c2682bd9d547fbb2821eff519c3e6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98d48ece99ec458b83525789b90cb939 (127.0.98.2:33811), 7f00fea834bf42a9b822e5750bcf6e23 (127.0.98.3:34819)
I20260928 02:38:10.603277 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:10.600732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40119 (local address 127.0.98.2:33811)
0928 02:38:10.600785 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:10.600787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:10.600798 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:38:10.600862 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:10.600863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:10.600874 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:10.600923 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:10.601080 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.601610 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.602631 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.602973 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.603029 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:10.603039 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:10.603177 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 02:38:10.603210 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 02:38:10.603307 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:10.600649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44075 (local address 127.0.98.3:34819)
0928 02:38:10.600712 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:10.600714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:10.600806 (+ 92us) server_negotiation.cc:407] Connection header received
0928 02:38:10.600861 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:10.600862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:10.600872 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:10.600920 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:10.601108 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.601617 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.602629 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.602980 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.603041 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:10.603050 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:10.603197 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 02:38:10.603234 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 02:38:10.603287 17872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:10.600576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33811 (local address 127.0.98.1:40119)
0928 02:38:10.600714 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:10.600725 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:10.600803 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:10.600934 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:10.600935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:10.600941 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:10.601018 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:10.601019 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.601683 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.601684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:10.602546 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:10.602548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.603047 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.603048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:10.603113 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:10.603117 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:38:10.603155 (+ 38us) client_negotiation.cc:240] Negotiation successful
0928 02:38:10.603194 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":56,"thread_start_us":65,"threads_started":1}
I20260928 02:38:10.603302 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:10.600576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34819 (local address 127.0.98.1:44075)
0928 02:38:10.600718 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:10.600735 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:10.600805 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:10.600934 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:10.600934 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:10.600937 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:10.601041 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:10.601043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.601690 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.601691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:10.602544 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:10.602546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.603055 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.603056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:10.603125 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:10.603128 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:38:10.603184 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 02:38:10.603219 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":54,"thread_start_us":83,"threads_started":1}
I20260928 02:38:10.603724 17666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4fd8cec03894cc9bba813d6298f77b6" candidate_uuid: "31c2682bd9d547fbb2821eff519c3e6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98d48ece99ec458b83525789b90cb939" is_pre_election: true
I20260928 02:38:10.603767 17800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4fd8cec03894cc9bba813d6298f77b6" candidate_uuid: "31c2682bd9d547fbb2821eff519c3e6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f00fea834bf42a9b822e5750bcf6e23" is_pre_election: true
I20260928 02:38:10.603892 17800 raft_consensus.cc:2454] T d4fd8cec03894cc9bba813d6298f77b6 P 7f00fea834bf42a9b822e5750bcf6e23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31c2682bd9d547fbb2821eff519c3e6e in term 0.
I20260928 02:38:10.603892 17666 raft_consensus.cc:2454] T d4fd8cec03894cc9bba813d6298f77b6 P 98d48ece99ec458b83525789b90cb939 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31c2682bd9d547fbb2821eff519c3e6e in term 0.
I20260928 02:38:10.604103 17467 leader_election.cc:302] T d4fd8cec03894cc9bba813d6298f77b6 P 31c2682bd9d547fbb2821eff519c3e6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31c2682bd9d547fbb2821eff519c3e6e, 7f00fea834bf42a9b822e5750bcf6e23; no voters:
I20260928 02:38:10.604224 17871 raft_consensus.cc:2781] T d4fd8cec03894cc9bba813d6298f77b6 P 31c2682bd9d547fbb2821eff519c3e6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:38:10.604271 17871 raft_consensus.cc:482] T d4fd8cec03894cc9bba813d6298f77b6 P 31c2682bd9d547fbb2821eff519c3e6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:38:10.604295 17871 raft_consensus.cc:3037] T d4fd8cec03894cc9bba813d6298f77b6 P 31c2682bd9d547fbb2821eff519c3e6e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:10.604949 17871 raft_consensus.cc:504] T d4fd8cec03894cc9bba813d6298f77b6 P 31c2682bd9d547fbb2821eff519c3e6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d48ece99ec458b83525789b90cb939" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33811 } } peers { permanent_uuid: "7f00fea834bf42a9b822e5750bcf6e23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34819 } } peers { permanent_uuid: "31c2682bd9d547fbb2821eff519c3e6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35945 } }
I20260928 02:38:10.605062 17871 leader_election.cc:288] T d4fd8cec03894cc9bba813d6298f77b6 P 31c2682bd9d547fbb2821eff519c3e6e [CANDIDATE]: Term 1 election: Requested vote from peers 98d48ece99ec458b83525789b90cb939 (127.0.98.2:33811), 7f00fea834bf42a9b822e5750bcf6e23 (127.0.98.3:34819)
I20260928 02:38:10.605214 17800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4fd8cec03894cc9bba813d6298f77b6" candidate_uuid: "31c2682bd9d547fbb2821eff519c3e6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f00fea834bf42a9b822e5750bcf6e23"
I20260928 02:38:10.605235 17666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4fd8cec03894cc9bba813d6298f77b6" candidate_uuid: "31c2682bd9d547fbb2821eff519c3e6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98d48ece99ec458b83525789b90cb939"
I20260928 02:38:10.605301 17800 raft_consensus.cc:3037] T d4fd8cec03894cc9bba813d6298f77b6 P 7f00fea834bf42a9b822e5750bcf6e23 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:10.605332 17666 raft_consensus.cc:3037] T d4fd8cec03894cc9bba813d6298f77b6 P 98d48ece99ec458b83525789b90cb939 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:10.605947 17800 raft_consensus.cc:2454] T d4fd8cec03894cc9bba813d6298f77b6 P 7f00fea834bf42a9b822e5750bcf6e23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31c2682bd9d547fbb2821eff519c3e6e in term 1.
I20260928 02:38:10.606055 17666 raft_consensus.cc:2454] T d4fd8cec03894cc9bba813d6298f77b6 P 98d48ece99ec458b83525789b90cb939 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31c2682bd9d547fbb2821eff519c3e6e in term 1.
I20260928 02:38:10.606109 17467 leader_election.cc:302] T d4fd8cec03894cc9bba813d6298f77b6 P 31c2682bd9d547fbb2821eff519c3e6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31c2682bd9d547fbb2821eff519c3e6e, 7f00fea834bf42a9b822e5750bcf6e23; no voters:
I20260928 02:38:10.606189 17871 raft_consensus.cc:2781] T d4fd8cec03894cc9bba813d6298f77b6 P 31c2682bd9d547fbb2821eff519c3e6e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:10.606348 17871 raft_consensus.cc:686] T d4fd8cec03894cc9bba813d6298f77b6 P 31c2682bd9d547fbb2821eff519c3e6e [term 1 LEADER]: Becoming Leader. State: Replica: 31c2682bd9d547fbb2821eff519c3e6e, State: Running, Role: LEADER
I20260928 02:38:10.606452 17871 consensus_queue.cc:237] T d4fd8cec03894cc9bba813d6298f77b6 P 31c2682bd9d547fbb2821eff519c3e6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d48ece99ec458b83525789b90cb939" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33811 } } peers { permanent_uuid: "7f00fea834bf42a9b822e5750bcf6e23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34819 } } peers { permanent_uuid: "31c2682bd9d547fbb2821eff519c3e6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35945 } }
I20260928 02:38:10.607213 17386 catalog_manager.cc:5495] T d4fd8cec03894cc9bba813d6298f77b6 P 31c2682bd9d547fbb2821eff519c3e6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 31c2682bd9d547fbb2821eff519c3e6e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "31c2682bd9d547fbb2821eff519c3e6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98d48ece99ec458b83525789b90cb939" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f00fea834bf42a9b822e5750bcf6e23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31c2682bd9d547fbb2821eff519c3e6e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35945 } health_report { overall_health: HEALTHY } } }
I20260928 02:38:10.621994 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:10.619838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34480 (local address 127.0.98.1:35945)
0928 02:38:10.619883 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:10.619884 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:10.619892 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:38:10.619954 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:10.619955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:10.619973 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:10.620025 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:38:10.620145 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.620590 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.621349 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.621531 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.621581 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:10.621724 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:38:10.621815 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:38:10.621881 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:10.621911 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 02:38:10.621928 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 02:38:10.623579 17800 raft_consensus.cc:1261] T d4fd8cec03894cc9bba813d6298f77b6 P 7f00fea834bf42a9b822e5750bcf6e23 [term 1 FOLLOWER]: Refusing update from remote peer 31c2682bd9d547fbb2821eff519c3e6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:38:10.623579 17666 raft_consensus.cc:1261] T d4fd8cec03894cc9bba813d6298f77b6 P 98d48ece99ec458b83525789b90cb939 [term 1 FOLLOWER]: Refusing update from remote peer 31c2682bd9d547fbb2821eff519c3e6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:38:10.623903 17874 consensus_queue.cc:1035] T d4fd8cec03894cc9bba813d6298f77b6 P 31c2682bd9d547fbb2821eff519c3e6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "98d48ece99ec458b83525789b90cb939" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:38:10.624027 17871 consensus_queue.cc:1035] T d4fd8cec03894cc9bba813d6298f77b6 P 31c2682bd9d547fbb2821eff519c3e6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f00fea834bf42a9b822e5750bcf6e23" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:38:10.628535 17386 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:38:10.628616 17386 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:38:10.629245 17386 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a4dbd81c7f114425a0de25212a16630b: Sending DeleteTablet for 3 replicas of tablet d4fd8cec03894cc9bba813d6298f77b6
I20260928 02:38:10.629617 17780 tablet_service.cc:1514] Processing DeleteTablet for tablet d4fd8cec03894cc9bba813d6298f77b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54082
I20260928 02:38:10.629832 17887 tablet_replica.cc:330] T d4fd8cec03894cc9bba813d6298f77b6 P 7f00fea834bf42a9b822e5750bcf6e23: stopping tablet replica
I20260928 02:38:10.629858 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 17449
I20260928 02:38:10.629889 17646 tablet_service.cc:1514] Processing DeleteTablet for tablet d4fd8cec03894cc9bba813d6298f77b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44712
I20260928 02:38:10.629953 17887 raft_consensus.cc:2229] T d4fd8cec03894cc9bba813d6298f77b6 P 7f00fea834bf42a9b822e5750bcf6e23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:38:10.630043 17887 raft_consensus.cc:2258] T d4fd8cec03894cc9bba813d6298f77b6 P 7f00fea834bf42a9b822e5750bcf6e23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:38:10.630439 17888 tablet_replica.cc:330] T d4fd8cec03894cc9bba813d6298f77b6 P 98d48ece99ec458b83525789b90cb939: stopping tablet replica
I20260928 02:38:10.630514 17888 raft_consensus.cc:2229] T d4fd8cec03894cc9bba813d6298f77b6 P 98d48ece99ec458b83525789b90cb939 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:38:10.630600 17888 raft_consensus.cc:2258] T d4fd8cec03894cc9bba813d6298f77b6 P 98d48ece99ec458b83525789b90cb939 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:38:10.630959 17888 ts_tablet_manager.cc:1905] T d4fd8cec03894cc9bba813d6298f77b6 P 98d48ece99ec458b83525789b90cb939: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:38:10.631322 17887 ts_tablet_manager.cc:1905] T d4fd8cec03894cc9bba813d6298f77b6 P 7f00fea834bf42a9b822e5750bcf6e23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:38:10.632526 17888 ts_tablet_manager.cc:1918] T d4fd8cec03894cc9bba813d6298f77b6 P 98d48ece99ec458b83525789b90cb939: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:38:10.632601 17888 log.cc:1196] T d4fd8cec03894cc9bba813d6298f77b6 P 98d48ece99ec458b83525789b90cb939: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-1/wal/wals/d4fd8cec03894cc9bba813d6298f77b6
I20260928 02:38:10.632756 17887 ts_tablet_manager.cc:1918] T d4fd8cec03894cc9bba813d6298f77b6 P 7f00fea834bf42a9b822e5750bcf6e23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:38:10.632823 17887 log.cc:1196] T d4fd8cec03894cc9bba813d6298f77b6 P 7f00fea834bf42a9b822e5750bcf6e23: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790562862804661-392-0/minicluster-data/ts-2/wal/wals/d4fd8cec03894cc9bba813d6298f77b6
I20260928 02:38:10.632870 17888 ts_tablet_manager.cc:1939] T d4fd8cec03894cc9bba813d6298f77b6 P 98d48ece99ec458b83525789b90cb939: Deleting consensus metadata
I20260928 02:38:10.633082 17887 ts_tablet_manager.cc:1939] T d4fd8cec03894cc9bba813d6298f77b6 P 7f00fea834bf42a9b822e5750bcf6e23: Deleting consensus metadata
I20260928 02:38:10.633503 17374 catalog_manager.cc:4841] TS 98d48ece99ec458b83525789b90cb939 (127.0.98.2:33811): tablet d4fd8cec03894cc9bba813d6298f77b6 (table test-table [id=7a8351ac71134ce5a4b7b0d860d6e24b]) successfully deleted
I20260928 02:38:10.633525 17373 catalog_manager.cc:4841] TS 7f00fea834bf42a9b822e5750bcf6e23 (127.0.98.3:34819): tablet d4fd8cec03894cc9bba813d6298f77b6 (table test-table [id=7a8351ac71134ce5a4b7b0d860d6e24b]) successfully deleted
W20260928 02:38:10.635524 17371 connection.cc:535] client connection to 127.0.98.1:35945 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 02:38:10.635609 17371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:38:10.635782 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 17583
I20260928 02:38:10.636405 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:10.635913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35945 (local address 127.0.0.1:34482)
0928 02:38:10.636346 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:10.636377 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415}
W20260928 02:38:10.636515 17371 catalog_manager.cc:4584] TS 31c2682bd9d547fbb2821eff519c3e6e (127.0.98.1:35945): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d4fd8cec03894cc9bba813d6298f77b6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35945: connect: Connection refused (error 111)
I20260928 02:38:10.640050 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 17717
I20260928 02:38:10.643746 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 17355
2026-09-28T02:38:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2072 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T02:38:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:38:10Z Disabled control of system clock
I20260928 02:38:10.667587 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33015
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:38:10.743558 17893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:10.743758 17893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:10.743777 17893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:10.744832 17893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:10.744865 17893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:10.744876 17893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:10.744887 17893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:10.744900 17893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:10.746166 17893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33015
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:10.746354 17893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:10.746579 17893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:10.750856 17898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:10.750979 17899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:10.751035 17901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:10.750936 17893 server_base.cc:1034] running on GCE node
I20260928 02:38:10.751310 17893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:10.751562 17893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:10.752698 17893 hybrid_clock.cc:648] HybridClock initialized: now 1790563090752683 us; error 30 us; skew 500 ppm
I20260928 02:38:10.753793 17893 webserver.cc:460] Webserver started at http://127.0.98.62:42803/ using document root <none> and password file <none>
I20260928 02:38:10.753993 17893 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:10.754040 17893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:10.754163 17893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:10.755120 17893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "8948124ca00c4514a4e9929615f579e9"
format_stamp: "Formatted at 2026-09-28 02:38:10 on dist-test-slave-fz47"
server_key: "5c8ad7cff68d50d3e9e9d9e771d1d1ff"
server_key_iv: "eedae210c415bdd6bd883e3adbd8ca9b"
server_key_version: "encryptionkey@0"
I20260928 02:38:10.755429 17893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "8948124ca00c4514a4e9929615f579e9"
format_stamp: "Formatted at 2026-09-28 02:38:10 on dist-test-slave-fz47"
server_key: "5c8ad7cff68d50d3e9e9d9e771d1d1ff"
server_key_iv: "eedae210c415bdd6bd883e3adbd8ca9b"
server_key_version: "encryptionkey@0"
I20260928 02:38:10.756548 17893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:38:10.757159 17907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:10.757333 17893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:10.757408 17893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "8948124ca00c4514a4e9929615f579e9"
format_stamp: "Formatted at 2026-09-28 02:38:10 on dist-test-slave-fz47"
server_key: "5c8ad7cff68d50d3e9e9d9e771d1d1ff"
server_key_iv: "eedae210c415bdd6bd883e3adbd8ca9b"
server_key_version: "encryptionkey@0"
I20260928 02:38:10.757477 17893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:10.794979 17893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:10.795297 17893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:10.795421 17893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:10.799326 17893 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33015
I20260928 02:38:10.799360 17959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33015 every 8 connection(s)
I20260928 02:38:10.799657 17893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:38:10.800545 17960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:10.801182 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 17893
I20260928 02:38:10.801290 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:38:10.801461 392 external_mini_cluster.cc:1428] Setting key 76a0fde5dca77af9c3c3f3cd5bfbfbd5
I20260928 02:38:10.803431 17960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8948124ca00c4514a4e9929615f579e9: Bootstrap starting.
I20260928 02:38:10.804081 17960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8948124ca00c4514a4e9929615f579e9: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:10.804361 17960 log.cc:824] T 00000000000000000000000000000000 P 8948124ca00c4514a4e9929615f579e9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:10.804965 17960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8948124ca00c4514a4e9929615f579e9: No bootstrap required, opened a new log
I20260928 02:38:10.807025 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:10.801915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57930 (local address 127.0.98.62:33015)
0928 02:38:10.802102 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:10.802111 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:10.802144 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:38:10.802647 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:10.802656 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:10.802739 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:10.802887 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:10.803070 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.804324 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.805047 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.805666 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.805801 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:10.806339 (+ 538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:10.806343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:10.806343 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:10.806535 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:10.806577 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:10.806593 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:10.806660 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:10.806715 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 02:38:10.806808 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
I20260928 02:38:10.807176 17960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8948124ca00c4514a4e9929615f579e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8948124ca00c4514a4e9929615f579e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33015 } }
I20260928 02:38:10.807308 17960 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8948124ca00c4514a4e9929615f579e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:10.807339 17960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8948124ca00c4514a4e9929615f579e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8948124ca00c4514a4e9929615f579e9, State: Initialized, Role: FOLLOWER
I20260928 02:38:10.807456 17960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8948124ca00c4514a4e9929615f579e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8948124ca00c4514a4e9929615f579e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33015 } }
I20260928 02:38:10.807523 17960 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8948124ca00c4514a4e9929615f579e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:38:10.807551 17960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8948124ca00c4514a4e9929615f579e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:38:10.807595 17960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8948124ca00c4514a4e9929615f579e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:10.808070 17960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8948124ca00c4514a4e9929615f579e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8948124ca00c4514a4e9929615f579e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33015 } }
I20260928 02:38:10.808152 17960 leader_election.cc:302] T 00000000000000000000000000000000 P 8948124ca00c4514a4e9929615f579e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8948124ca00c4514a4e9929615f579e9; no voters:
I20260928 02:38:10.808279 17960 leader_election.cc:288] T 00000000000000000000000000000000 P 8948124ca00c4514a4e9929615f579e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:38:10.808316 17965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8948124ca00c4514a4e9929615f579e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:10.808491 17960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8948124ca00c4514a4e9929615f579e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:38:10.808478 17965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8948124ca00c4514a4e9929615f579e9 [term 1 LEADER]: Becoming Leader. State: Replica: 8948124ca00c4514a4e9929615f579e9, State: Running, Role: LEADER
I20260928 02:38:10.808606 17965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8948124ca00c4514a4e9929615f579e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8948124ca00c4514a4e9929615f579e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33015 } }
I20260928 02:38:10.808941 17967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8948124ca00c4514a4e9929615f579e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8948124ca00c4514a4e9929615f579e9. Latest consensus state: current_term: 1 leader_uuid: "8948124ca00c4514a4e9929615f579e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8948124ca00c4514a4e9929615f579e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33015 } } }
I20260928 02:38:10.808924 17966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8948124ca00c4514a4e9929615f579e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8948124ca00c4514a4e9929615f579e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8948124ca00c4514a4e9929615f579e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33015 } } }
I20260928 02:38:10.809183 17966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8948124ca00c4514a4e9929615f579e9 [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:10.809121 17967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8948124ca00c4514a4e9929615f579e9 [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:10.809490 17974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:38:10.809970 17974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:38:10.811365 17974 catalog_manager.cc:1347] Generated new cluster ID: bfb7433601394fb48474d9f9f9680bdc
I20260928 02:38:10.811405 17974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:38:10.817174 17974 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:38:10.817673 17974 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:38:10.821539 17974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8948124ca00c4514a4e9929615f579e9: Generated new TSK 0
I20260928 02:38:10.821696 17974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:38:10.830752 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33015
--builtin_ntp_servers=127.0.98.20:37089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:38:10.916215 17984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:10.916522 17984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:10.916580 17984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:10.918273 17984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:10.918360 17984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:10.918424 17984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:38:10.919839 17984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:10.920068 17984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:10.920361 17984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:10.925752 17990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:10.925772 17989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:10.926096 17992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:10.926244 17984 server_base.cc:1034] running on GCE node
I20260928 02:38:10.926481 17984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:10.926761 17984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:10.927897 17984 hybrid_clock.cc:648] HybridClock initialized: now 1790563090927872 us; error 37 us; skew 500 ppm
I20260928 02:38:10.929368 17984 webserver.cc:460] Webserver started at http://127.0.98.1:44977/ using document root <none> and password file <none>
I20260928 02:38:10.929622 17984 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:10.929682 17984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:10.929803 17984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:10.931034 17984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-0/data/instance:
uuid: "2f718b17cefd4dbfb129aa3de77d4510"
format_stamp: "Formatted at 2026-09-28 02:38:10 on dist-test-slave-fz47"
server_key: "fa1bcd337afd02bddc829f76894826ca"
server_key_iv: "fe22bc3057b6024d93e67ab0310afeff"
server_key_version: "encryptionkey@0"
I20260928 02:38:10.931414 17984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2f718b17cefd4dbfb129aa3de77d4510"
format_stamp: "Formatted at 2026-09-28 02:38:10 on dist-test-slave-fz47"
server_key: "fa1bcd337afd02bddc829f76894826ca"
server_key_iv: "fe22bc3057b6024d93e67ab0310afeff"
server_key_version: "encryptionkey@0"
I20260928 02:38:10.932950 17984 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:38:10.933764 17998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:10.933918 17984 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:10.933987 17984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "2f718b17cefd4dbfb129aa3de77d4510"
format_stamp: "Formatted at 2026-09-28 02:38:10 on dist-test-slave-fz47"
server_key: "fa1bcd337afd02bddc829f76894826ca"
server_key_iv: "fe22bc3057b6024d93e67ab0310afeff"
server_key_version: "encryptionkey@0"
I20260928 02:38:10.934052 17984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:10.951431 17984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:10.951682 17984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:10.951802 17984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:10.952000 17984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:38:10.952298 17984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:38:10.952329 17984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:10.952356 17984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:38:10.952381 17984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:10.957667 17984 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37117
I20260928 02:38:10.957712 18111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37117 every 8 connection(s)
I20260928 02:38:10.958011 17984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
I20260928 02:38:10.962875 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:10.958634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55785 (local address 127.0.98.62:33015)
0928 02:38:10.958691 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:10.958693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:10.958930 (+ 237us) server_negotiation.cc:407] Connection header received
0928 02:38:10.959421 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:10.959422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:10.959437 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:10.959485 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:10.960175 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.960673 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.961679 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.961890 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.961946 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:10.962620 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:10.962621 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:10.962622 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:10.962626 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:10.962642 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:10.962646 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:10.962691 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:10.962767 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 02:38:10.962798 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 02:38:10.963038 18114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:10.958619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33015 (local address 127.0.98.1:55785)
0928 02:38:10.958818 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:10.958850 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:10.959332 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:10.959562 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:10.959565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:10.959574 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:10.960095 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:10.960100 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.960772 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.960773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:10.961603 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:10.961607 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:10.961960 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:10.961961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:10.962078 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:10.962498 (+ 420us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:38:10.962499 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:10.962525 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:38:10.962546 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:10.962696 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:10.962701 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:10.962704 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:38:10.962760 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 02:38:10.962858 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":100,"threads_started":1}
I20260928 02:38:10.963445 18112 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33015
I20260928 02:38:10.963522 18112 heartbeater.cc:464] Registering TS with master...
I20260928 02:38:10.963683 18112 heartbeater.cc:511] Master 127.0.98.62:33015 requested a full tablet report, sending...
I20260928 02:38:10.964088 17924 ts_manager.cc:194] Registered new tserver with Master: 2f718b17cefd4dbfb129aa3de77d4510 (127.0.98.1:37117)
I20260928 02:38:10.964697 17924 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:55785
I20260928 02:38:10.965423 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 17984
I20260928 02:38:10.965505 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-0/wal/instance
I20260928 02:38:10.965680 392 external_mini_cluster.cc:1428] Setting key d031e71950d72897f6a8b55ca3620ce0
I20260928 02:38:10.966444 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33015
--builtin_ntp_servers=127.0.98.20:37089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:38:11.042578 18115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:11.042764 18115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:11.042790 18115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:11.043943 18115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:11.043991 18115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:11.044020 18115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:38:11.045454 18115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:11.045639 18115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:11.045861 18115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:11.050237 18120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:11.050294 18115 server_base.cc:1034] running on GCE node
W20260928 02:38:11.050416 18121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:11.050328 18123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:11.050688 18115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:11.050877 18115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:11.052004 18115 hybrid_clock.cc:648] HybridClock initialized: now 1790563091051993 us; error 29 us; skew 500 ppm
I20260928 02:38:11.053114 18115 webserver.cc:460] Webserver started at http://127.0.98.2:45831/ using document root <none> and password file <none>
I20260928 02:38:11.053357 18115 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:11.053403 18115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:11.053522 18115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:11.054507 18115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-1/data/instance:
uuid: "ae3bd0bc51b04088ad8cf4872276d9a5"
format_stamp: "Formatted at 2026-09-28 02:38:11 on dist-test-slave-fz47"
server_key: "40b42246c3a4f9360ac4be23fc5ca17a"
server_key_iv: "e270bc805ffb873730c062b7beb118d9"
server_key_version: "encryptionkey@0"
I20260928 02:38:11.054823 18115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ae3bd0bc51b04088ad8cf4872276d9a5"
format_stamp: "Formatted at 2026-09-28 02:38:11 on dist-test-slave-fz47"
server_key: "40b42246c3a4f9360ac4be23fc5ca17a"
server_key_iv: "e270bc805ffb873730c062b7beb118d9"
server_key_version: "encryptionkey@0"
I20260928 02:38:11.055931 18115 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 02:38:11.056532 18129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:11.056653 18115 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:38:11.056723 18115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "ae3bd0bc51b04088ad8cf4872276d9a5"
format_stamp: "Formatted at 2026-09-28 02:38:11 on dist-test-slave-fz47"
server_key: "40b42246c3a4f9360ac4be23fc5ca17a"
server_key_iv: "e270bc805ffb873730c062b7beb118d9"
server_key_version: "encryptionkey@0"
I20260928 02:38:11.056787 18115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:11.084532 18115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:11.084831 18115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:11.084955 18115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:11.085177 18115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:38:11.085533 18115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:38:11.085567 18115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:11.085596 18115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:38:11.085618 18115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:11.090741 18115 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36675
I20260928 02:38:11.090812 18242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36675 every 8 connection(s)
I20260928 02:38:11.091080 18115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
I20260928 02:38:11.095813 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:11.091883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38339 (local address 127.0.98.62:33015)
0928 02:38:11.091930 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:11.091932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:11.092145 (+ 213us) server_negotiation.cc:407] Connection header received
0928 02:38:11.092613 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:11.092614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:11.092624 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:11.092674 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:11.093305 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:11.093799 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:11.094707 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:11.094886 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:11.094927 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:11.095627 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:11.095628 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:11.095628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:11.095631 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:11.095641 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:11.095643 (+ 2us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:11.095671 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:11.095731 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 02:38:11.095754 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:38:11.095988 18245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:11.091878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33015 (local address 127.0.98.2:38339)
0928 02:38:11.092040 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:11.092074 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:11.092526 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:11.092735 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:11.092738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:11.092747 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:11.093225 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:11.093229 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:11.093859 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:11.093859 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:11.094639 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:11.094641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:11.094934 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:11.094935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:11.095035 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:11.095518 (+ 483us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:38:11.095519 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:11.095546 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:38:11.095568 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:11.095680 (+ 112us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:11.095683 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:11.095685 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:38:11.095727 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 02:38:11.095800 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
I20260928 02:38:11.096400 18243 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33015
I20260928 02:38:11.096474 18243 heartbeater.cc:464] Registering TS with master...
I20260928 02:38:11.096611 18243 heartbeater.cc:511] Master 127.0.98.62:33015 requested a full tablet report, sending...
I20260928 02:38:11.096936 17924 ts_manager.cc:194] Registered new tserver with Master: ae3bd0bc51b04088ad8cf4872276d9a5 (127.0.98.2:36675)
I20260928 02:38:11.097292 17924 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:38339
I20260928 02:38:11.100780 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 18115
I20260928 02:38:11.100867 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-1/wal/instance
I20260928 02:38:11.101033 392 external_mini_cluster.cc:1428] Setting key 6a9e086ce98ed31c20ee9409d6768b50
I20260928 02:38:11.101843 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33015
--builtin_ntp_servers=127.0.98.20:37089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:38:11.179140 18246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:11.179320 18246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:11.179339 18246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:11.180418 18246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:11.180454 18246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:11.180477 18246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:38:11.181833 18246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:11.182035 18246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:11.182296 18246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:11.186896 18254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:11.186908 18252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:11.187042 18246 server_base.cc:1034] running on GCE node
W20260928 02:38:11.186930 18251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:11.187333 18246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:11.187533 18246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:11.188668 18246 hybrid_clock.cc:648] HybridClock initialized: now 1790563091188658 us; error 26 us; skew 500 ppm
I20260928 02:38:11.189906 18246 webserver.cc:460] Webserver started at http://127.0.98.3:36145/ using document root <none> and password file <none>
I20260928 02:38:11.190127 18246 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:11.190172 18246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:11.190296 18246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:11.191341 18246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-2/data/instance:
uuid: "fd9b613ccb15437daa7d40397e9ab3a7"
format_stamp: "Formatted at 2026-09-28 02:38:11 on dist-test-slave-fz47"
server_key: "222bdee7d5f23829cd96636f1b826c37"
server_key_iv: "8289b28f850cc8fcd9eecc2db5c54899"
server_key_version: "encryptionkey@0"
I20260928 02:38:11.191665 18246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fd9b613ccb15437daa7d40397e9ab3a7"
format_stamp: "Formatted at 2026-09-28 02:38:11 on dist-test-slave-fz47"
server_key: "222bdee7d5f23829cd96636f1b826c37"
server_key_iv: "8289b28f850cc8fcd9eecc2db5c54899"
server_key_version: "encryptionkey@0"
I20260928 02:38:11.192876 18246 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:38:11.193555 18260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:11.193704 18246 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:11.193783 18246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "fd9b613ccb15437daa7d40397e9ab3a7"
format_stamp: "Formatted at 2026-09-28 02:38:11 on dist-test-slave-fz47"
server_key: "222bdee7d5f23829cd96636f1b826c37"
server_key_iv: "8289b28f850cc8fcd9eecc2db5c54899"
server_key_version: "encryptionkey@0"
I20260928 02:38:11.193843 18246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:11.237071 18246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:11.237401 18246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:11.237527 18246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:11.237769 18246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:38:11.238106 18246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:38:11.238138 18246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:11.238168 18246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:38:11.238184 18246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:11.243196 18246 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46845
I20260928 02:38:11.243257 18373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46845 every 8 connection(s)
I20260928 02:38:11.243546 18246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
I20260928 02:38:11.246344 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 18246
I20260928 02:38:11.246445 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-2/wal/instance
I20260928 02:38:11.246604 392 external_mini_cluster.cc:1428] Setting key 0801f4cdffd81203e7bc494531a8461d
I20260928 02:38:11.248799 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:11.244223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36529 (local address 127.0.98.62:33015)
0928 02:38:11.244291 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:11.244293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:11.244521 (+ 228us) server_negotiation.cc:407] Connection header received
0928 02:38:11.245024 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:11.245025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:11.245037 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:11.245101 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:11.245818 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:11.246321 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:11.247590 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:11.247775 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:11.247827 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:11.248553 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:11.248554 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:11.248554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:11.248558 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:11.248573 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:11.248585 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:11.248627 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:11.248698 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 02:38:11.248722 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 02:38:11.248996 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:11.244198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33015 (local address 127.0.98.3:36529)
0928 02:38:11.244406 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:11.244440 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:11.244928 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:11.245195 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:11.245199 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:11.245207 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:11.245733 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:11.245738 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:11.246405 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:11.246406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:11.247509 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:11.247512 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:11.247843 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:11.247844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:11.247975 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:11.248432 (+ 457us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:38:11.248432 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:11.248460 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:38:11.248482 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:11.248633 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:11.248638 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:11.248641 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:38:11.248701 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 02:38:11.248800 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":101,"threads_started":1}
I20260928 02:38:11.249476 18374 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33015
I20260928 02:38:11.249552 18374 heartbeater.cc:464] Registering TS with master...
I20260928 02:38:11.249696 18374 heartbeater.cc:511] Master 127.0.98.62:33015 requested a full tablet report, sending...
I20260928 02:38:11.249994 17924 ts_manager.cc:194] Registered new tserver with Master: fd9b613ccb15437daa7d40397e9ab3a7 (127.0.98.3:46845)
I20260928 02:38:11.250352 17924 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:36529
I20260928 02:38:11.258198 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:38:11.262303 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:11.259428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57932 (local address 127.0.98.62:33015)
0928 02:38:11.259480 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:11.259482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:11.259610 (+ 128us) server_negotiation.cc:407] Connection header received
0928 02:38:11.259650 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:11.259651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:11.259660 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:11.259705 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:11.259835 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:11.260386 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:11.261139 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:11.261360 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:11.261416 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:11.262086 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:11.262087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:11.262088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:11.262091 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:11.262112 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:11.262120 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:11.262169 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:11.262208 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 02:38:11.262229 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 02:38:11.263810 17924 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:38:11.265316 17924 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:38:11.271265 18384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:11.267400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36675 (local address 127.0.0.1:43836)
0928 02:38:11.267573 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:11.267614 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:11.267690 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:11.268050 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:11.268051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:11.268053 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:11.268107 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:11.268108 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:11.269421 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:11.269422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:11.270379 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:11.270381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:11.270978 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:11.270979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:11.271052 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:11.271065 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 02:38:11.271128 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 02:38:11.271182 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":1,"thread_start_us":60,"threads_started":1}
I20260928 02:38:11.271270 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:11.267569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43836 (local address 127.0.98.2:36675)
0928 02:38:11.267832 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:11.267837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:11.267846 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:38:11.267888 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:11.267891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:11.267906 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:11.268048 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:11.268185 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:11.269337 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:11.270450 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:11.270907 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:11.270968 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:11.271056 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:11.271155 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 02:38:11.271192 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":87,"threads_started":1}
I20260928 02:38:11.271492 18382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:11.267401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37117 (local address 127.0.0.1:55246)
0928 02:38:11.267741 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:11.267751 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:11.267803 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:11.267967 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:11.267970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:11.267972 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:11.268034 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:11.268036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:11.268974 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:11.268975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:11.270144 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:11.270147 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:11.271146 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:11.271147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:11.271205 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:11.271209 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:38:11.271251 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 02:38:11.271290 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":1,"thread_start_us":61,"threads_started":1}
I20260928 02:38:11.271782 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:11.267407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46845 (local address 127.0.0.1:51378)
0928 02:38:11.267872 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:11.267880 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:11.267913 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:11.268294 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:11.268295 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:11.268296 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:11.268330 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:11.268332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:11.269513 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:11.269514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:11.270402 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:11.270405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:11.271334 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:11.271335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:11.271633 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:11.271636 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:38:11.271686 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 02:38:11.271721 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":86,"threads_started":1}
I20260928 02:38:11.271796 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:11.267639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55246 (local address 127.0.98.1:37117)
0928 02:38:11.267820 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:11.267824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:11.267834 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:38:11.267866 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:11.267868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:11.267879 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:11.267944 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:11.268106 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:11.268906 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:11.270220 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:11.270961 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:11.271027 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:11.271111 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:11.271657 (+ 546us) server_negotiation.cc:299] Negotiation successful
0928 02:38:11.271704 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":79,"threads_started":1}
I20260928 02:38:11.271976 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:11.267630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51378 (local address 127.0.98.3:46845)
0928 02:38:11.268162 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:11.268166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:11.268175 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:38:11.268207 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:11.268209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:11.268224 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:11.268295 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:11.268392 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:11.269448 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:11.270489 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:11.271161 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:11.271215 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:11.271301 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:11.271881 (+ 580us) server_negotiation.cc:299] Negotiation successful
0928 02:38:11.271922 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":72,"threads_started":1}
I20260928 02:38:11.271950 18177 tablet_service.cc:1467] Processing CreateTablet for tablet 48dacca0481d45e99ea3c3c66487fa99 (DEFAULT_TABLE table=test-table [id=128eb2ff4e1f4a43af59b3e15156e272]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:11.272301 18177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48dacca0481d45e99ea3c3c66487fa99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:11.272469 18046 tablet_service.cc:1467] Processing CreateTablet for tablet 48dacca0481d45e99ea3c3c66487fa99 (DEFAULT_TABLE table=test-table [id=128eb2ff4e1f4a43af59b3e15156e272]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:11.272756 18046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48dacca0481d45e99ea3c3c66487fa99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:11.274859 18388 tablet_bootstrap.cc:492] T 48dacca0481d45e99ea3c3c66487fa99 P ae3bd0bc51b04088ad8cf4872276d9a5: Bootstrap starting.
I20260928 02:38:11.275236 18389 tablet_bootstrap.cc:492] T 48dacca0481d45e99ea3c3c66487fa99 P 2f718b17cefd4dbfb129aa3de77d4510: Bootstrap starting.
I20260928 02:38:11.275452 18308 tablet_service.cc:1467] Processing CreateTablet for tablet 48dacca0481d45e99ea3c3c66487fa99 (DEFAULT_TABLE table=test-table [id=128eb2ff4e1f4a43af59b3e15156e272]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:11.275701 18388 tablet_bootstrap.cc:654] T 48dacca0481d45e99ea3c3c66487fa99 P ae3bd0bc51b04088ad8cf4872276d9a5: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:11.275790 18389 tablet_bootstrap.cc:654] T 48dacca0481d45e99ea3c3c66487fa99 P 2f718b17cefd4dbfb129aa3de77d4510: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:11.275789 18308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48dacca0481d45e99ea3c3c66487fa99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:11.275998 18388 log.cc:824] T 48dacca0481d45e99ea3c3c66487fa99 P ae3bd0bc51b04088ad8cf4872276d9a5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:11.276043 18389 log.cc:824] T 48dacca0481d45e99ea3c3c66487fa99 P 2f718b17cefd4dbfb129aa3de77d4510: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:11.276839 18389 tablet_bootstrap.cc:492] T 48dacca0481d45e99ea3c3c66487fa99 P 2f718b17cefd4dbfb129aa3de77d4510: No bootstrap required, opened a new log
I20260928 02:38:11.276899 18389 ts_tablet_manager.cc:1397] T 48dacca0481d45e99ea3c3c66487fa99 P 2f718b17cefd4dbfb129aa3de77d4510: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:38:11.277842 18388 tablet_bootstrap.cc:492] T 48dacca0481d45e99ea3c3c66487fa99 P ae3bd0bc51b04088ad8cf4872276d9a5: No bootstrap required, opened a new log
I20260928 02:38:11.277912 18388 ts_tablet_manager.cc:1397] T 48dacca0481d45e99ea3c3c66487fa99 P ae3bd0bc51b04088ad8cf4872276d9a5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 02:38:11.278409 18392 tablet_bootstrap.cc:492] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7: Bootstrap starting.
I20260928 02:38:11.278913 18392 tablet_bootstrap.cc:654] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:11.279157 18392 log.cc:824] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:11.279264 18389 raft_consensus.cc:348] T 48dacca0481d45e99ea3c3c66487fa99 P 2f718b17cefd4dbfb129aa3de77d4510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae3bd0bc51b04088ad8cf4872276d9a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36675 } } peers { permanent_uuid: "fd9b613ccb15437daa7d40397e9ab3a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46845 } } peers { permanent_uuid: "2f718b17cefd4dbfb129aa3de77d4510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37117 } }
I20260928 02:38:11.279433 18389 raft_consensus.cc:374] T 48dacca0481d45e99ea3c3c66487fa99 P 2f718b17cefd4dbfb129aa3de77d4510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:11.279475 18389 raft_consensus.cc:729] T 48dacca0481d45e99ea3c3c66487fa99 P 2f718b17cefd4dbfb129aa3de77d4510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f718b17cefd4dbfb129aa3de77d4510, State: Initialized, Role: FOLLOWER
I20260928 02:38:11.279600 18389 consensus_queue.cc:260] T 48dacca0481d45e99ea3c3c66487fa99 P 2f718b17cefd4dbfb129aa3de77d4510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae3bd0bc51b04088ad8cf4872276d9a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36675 } } peers { permanent_uuid: "fd9b613ccb15437daa7d40397e9ab3a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46845 } } peers { permanent_uuid: "2f718b17cefd4dbfb129aa3de77d4510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37117 } }
I20260928 02:38:11.279846 18389 ts_tablet_manager.cc:1428] T 48dacca0481d45e99ea3c3c66487fa99 P 2f718b17cefd4dbfb129aa3de77d4510: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:38:11.279881 18112 heartbeater.cc:503] Master 127.0.98.62:33015 was elected leader, sending a full tablet report...
I20260928 02:38:11.280195 18392 tablet_bootstrap.cc:492] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7: No bootstrap required, opened a new log
I20260928 02:38:11.280261 18392 ts_tablet_manager.cc:1397] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:38:11.280295 18388 raft_consensus.cc:348] T 48dacca0481d45e99ea3c3c66487fa99 P ae3bd0bc51b04088ad8cf4872276d9a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae3bd0bc51b04088ad8cf4872276d9a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36675 } } peers { permanent_uuid: "fd9b613ccb15437daa7d40397e9ab3a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46845 } } peers { permanent_uuid: "2f718b17cefd4dbfb129aa3de77d4510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37117 } }
I20260928 02:38:11.280431 18388 raft_consensus.cc:374] T 48dacca0481d45e99ea3c3c66487fa99 P ae3bd0bc51b04088ad8cf4872276d9a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:11.280465 18388 raft_consensus.cc:729] T 48dacca0481d45e99ea3c3c66487fa99 P ae3bd0bc51b04088ad8cf4872276d9a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae3bd0bc51b04088ad8cf4872276d9a5, State: Initialized, Role: FOLLOWER
I20260928 02:38:11.280572 18388 consensus_queue.cc:260] T 48dacca0481d45e99ea3c3c66487fa99 P ae3bd0bc51b04088ad8cf4872276d9a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae3bd0bc51b04088ad8cf4872276d9a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36675 } } peers { permanent_uuid: "fd9b613ccb15437daa7d40397e9ab3a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46845 } } peers { permanent_uuid: "2f718b17cefd4dbfb129aa3de77d4510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37117 } }
I20260928 02:38:11.280808 18388 ts_tablet_manager.cc:1428] T 48dacca0481d45e99ea3c3c66487fa99 P ae3bd0bc51b04088ad8cf4872276d9a5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:38:11.280944 18243 heartbeater.cc:503] Master 127.0.98.62:33015 was elected leader, sending a full tablet report...
I20260928 02:38:11.282262 18392 raft_consensus.cc:348] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae3bd0bc51b04088ad8cf4872276d9a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36675 } } peers { permanent_uuid: "fd9b613ccb15437daa7d40397e9ab3a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46845 } } peers { permanent_uuid: "2f718b17cefd4dbfb129aa3de77d4510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37117 } }
I20260928 02:38:11.282375 18392 raft_consensus.cc:374] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:11.282394 18392 raft_consensus.cc:729] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd9b613ccb15437daa7d40397e9ab3a7, State: Initialized, Role: FOLLOWER
I20260928 02:38:11.282466 18392 consensus_queue.cc:260] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae3bd0bc51b04088ad8cf4872276d9a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36675 } } peers { permanent_uuid: "fd9b613ccb15437daa7d40397e9ab3a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46845 } } peers { permanent_uuid: "2f718b17cefd4dbfb129aa3de77d4510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37117 } }
I20260928 02:38:11.282668 18392 ts_tablet_manager.cc:1428] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 02:38:11.282720 18374 heartbeater.cc:503] Master 127.0.98.62:33015 was elected leader, sending a full tablet report...
I20260928 02:38:11.297719 18396 raft_consensus.cc:482] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:38:11.297813 18396 raft_consensus.cc:504] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae3bd0bc51b04088ad8cf4872276d9a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36675 } } peers { permanent_uuid: "fd9b613ccb15437daa7d40397e9ab3a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46845 } } peers { permanent_uuid: "2f718b17cefd4dbfb129aa3de77d4510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37117 } }
I20260928 02:38:11.298115 18396 leader_election.cc:288] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae3bd0bc51b04088ad8cf4872276d9a5 (127.0.98.2:36675), 2f718b17cefd4dbfb129aa3de77d4510 (127.0.98.1:37117)
I20260928 02:38:11.300595 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:11.298336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54463 (local address 127.0.98.1:37117)
0928 02:38:11.298376 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:11.298379 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:11.298386 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:38:11.298413 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:11.298413 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:11.298424 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:11.298475 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:11.298606 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:11.299080 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:11.299973 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:11.300322 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:11.300379 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:11.300388 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:11.300501 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 02:38:11.300533 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:38:11.300597 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:11.298267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37117 (local address 127.0.98.3:54463)
0928 02:38:11.298315 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:11.298327 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:11.298363 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:11.298491 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:11.298492 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:11.298493 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:11.298552 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:11.298554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:11.299151 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:11.299152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:11.299896 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:11.299898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:11.300385 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:11.300386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:11.300446 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:11.300449 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:38:11.300491 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 02:38:11.300530 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":27}
I20260928 02:38:11.300693 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:11.298345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52367 (local address 127.0.98.2:36675)
0928 02:38:11.298384 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:11.298386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:11.298511 (+ 125us) server_negotiation.cc:407] Connection header received
0928 02:38:11.298554 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:11.298555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:11.298565 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:11.298609 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:11.298734 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:11.299180 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:11.300063 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:11.300406 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:11.300462 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:11.300471 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:11.300602 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 02:38:11.300633 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:38:11.300709 18397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:11.298268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36675 (local address 127.0.98.3:52367)
0928 02:38:11.298461 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:11.298468 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:11.298504 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:11.298616 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:11.298617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:11.298618 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:11.298682 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:11.298683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:11.299246 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:11.299247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:11.299986 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:11.299988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:11.300471 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:11.300472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:11.300547 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:11.300550 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:38:11.300592 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 02:38:11.300626 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":67,"thread_start_us":87,"threads_started":1}
I20260928 02:38:11.300889 18066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48dacca0481d45e99ea3c3c66487fa99" candidate_uuid: "fd9b613ccb15437daa7d40397e9ab3a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f718b17cefd4dbfb129aa3de77d4510" is_pre_election: true
I20260928 02:38:11.300976 18197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48dacca0481d45e99ea3c3c66487fa99" candidate_uuid: "fd9b613ccb15437daa7d40397e9ab3a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae3bd0bc51b04088ad8cf4872276d9a5" is_pre_election: true
I20260928 02:38:11.301023 18066 raft_consensus.cc:2454] T 48dacca0481d45e99ea3c3c66487fa99 P 2f718b17cefd4dbfb129aa3de77d4510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd9b613ccb15437daa7d40397e9ab3a7 in term 0.
I20260928 02:38:11.301108 18197 raft_consensus.cc:2454] T 48dacca0481d45e99ea3c3c66487fa99 P ae3bd0bc51b04088ad8cf4872276d9a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd9b613ccb15437daa7d40397e9ab3a7 in term 0.
I20260928 02:38:11.301203 18261 leader_election.cc:302] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f718b17cefd4dbfb129aa3de77d4510, fd9b613ccb15437daa7d40397e9ab3a7; no voters:
I20260928 02:38:11.301362 18396 raft_consensus.cc:2781] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:38:11.301412 18396 raft_consensus.cc:482] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:38:11.301427 18396 raft_consensus.cc:3037] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:11.302023 18396 raft_consensus.cc:504] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae3bd0bc51b04088ad8cf4872276d9a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36675 } } peers { permanent_uuid: "fd9b613ccb15437daa7d40397e9ab3a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46845 } } peers { permanent_uuid: "2f718b17cefd4dbfb129aa3de77d4510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37117 } }
I20260928 02:38:11.302124 18396 leader_election.cc:288] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7 [CANDIDATE]: Term 1 election: Requested vote from peers ae3bd0bc51b04088ad8cf4872276d9a5 (127.0.98.2:36675), 2f718b17cefd4dbfb129aa3de77d4510 (127.0.98.1:37117)
I20260928 02:38:11.302265 18197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48dacca0481d45e99ea3c3c66487fa99" candidate_uuid: "fd9b613ccb15437daa7d40397e9ab3a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae3bd0bc51b04088ad8cf4872276d9a5"
I20260928 02:38:11.302328 18197 raft_consensus.cc:3037] T 48dacca0481d45e99ea3c3c66487fa99 P ae3bd0bc51b04088ad8cf4872276d9a5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:11.302351 18066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48dacca0481d45e99ea3c3c66487fa99" candidate_uuid: "fd9b613ccb15437daa7d40397e9ab3a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f718b17cefd4dbfb129aa3de77d4510"
I20260928 02:38:11.302425 18066 raft_consensus.cc:3037] T 48dacca0481d45e99ea3c3c66487fa99 P 2f718b17cefd4dbfb129aa3de77d4510 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:11.303013 18197 raft_consensus.cc:2454] T 48dacca0481d45e99ea3c3c66487fa99 P ae3bd0bc51b04088ad8cf4872276d9a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd9b613ccb15437daa7d40397e9ab3a7 in term 1.
I20260928 02:38:11.303104 18066 raft_consensus.cc:2454] T 48dacca0481d45e99ea3c3c66487fa99 P 2f718b17cefd4dbfb129aa3de77d4510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd9b613ccb15437daa7d40397e9ab3a7 in term 1.
I20260928 02:38:11.303167 18263 leader_election.cc:302] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae3bd0bc51b04088ad8cf4872276d9a5, fd9b613ccb15437daa7d40397e9ab3a7; no voters:
I20260928 02:38:11.303298 18396 raft_consensus.cc:2781] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:11.303473 18396 raft_consensus.cc:686] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7 [term 1 LEADER]: Becoming Leader. State: Replica: fd9b613ccb15437daa7d40397e9ab3a7, State: Running, Role: LEADER
I20260928 02:38:11.303553 18396 consensus_queue.cc:237] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae3bd0bc51b04088ad8cf4872276d9a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36675 } } peers { permanent_uuid: "fd9b613ccb15437daa7d40397e9ab3a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46845 } } peers { permanent_uuid: "2f718b17cefd4dbfb129aa3de77d4510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37117 } }
I20260928 02:38:11.304263 17923 catalog_manager.cc:5495] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd9b613ccb15437daa7d40397e9ab3a7 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "fd9b613ccb15437daa7d40397e9ab3a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae3bd0bc51b04088ad8cf4872276d9a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd9b613ccb15437daa7d40397e9ab3a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f718b17cefd4dbfb129aa3de77d4510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37117 } health_report { overall_health: UNKNOWN } } }
I20260928 02:38:11.316617 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:11.314539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51384 (local address 127.0.98.3:46845)
0928 02:38:11.314578 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:11.314580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:11.314588 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:38:11.314637 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:11.314638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:11.314654 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:11.314713 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:38:11.314828 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:11.315275 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:11.315955 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:11.316151 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:11.316202 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:11.316328 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:38:11.316427 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:38:11.316495 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:11.316514 (+ 19us) server_negotiation.cc:299] Negotiation successful
0928 02:38:11.316534 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:38:11.318167 18066 raft_consensus.cc:1261] T 48dacca0481d45e99ea3c3c66487fa99 P 2f718b17cefd4dbfb129aa3de77d4510 [term 1 FOLLOWER]: Refusing update from remote peer fd9b613ccb15437daa7d40397e9ab3a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:38:11.318173 18197 raft_consensus.cc:1261] T 48dacca0481d45e99ea3c3c66487fa99 P ae3bd0bc51b04088ad8cf4872276d9a5 [term 1 FOLLOWER]: Refusing update from remote peer fd9b613ccb15437daa7d40397e9ab3a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:38:11.318460 18396 consensus_queue.cc:1035] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae3bd0bc51b04088ad8cf4872276d9a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:38:11.318572 18398 consensus_queue.cc:1035] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f718b17cefd4dbfb129aa3de77d4510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:38:11.320641 18401 mvcc.cc:205] Tried to move back new op lower bound from 7334146422037094400 to 7334146421979992064. Current Snapshot: MvccSnapshot[applied={T|T < 7334146422037094400}]
I20260928 02:38:11.321156 18403 mvcc.cc:205] Tried to move back new op lower bound from 7334146422037094400 to 7334146421979992064. Current Snapshot: MvccSnapshot[applied={T|T < 7334146422037094400 or (T in {7334146422037094400})}]
I20260928 02:38:11.321784 18402 mvcc.cc:205] Tried to move back new op lower bound from 7334146422037094400 to 7334146421979992064. Current Snapshot: MvccSnapshot[applied={T|T < 7334146422037094400}]
I20260928 02:38:11.322751 17924 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:38:11.322821 17924 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:38:11.323448 17924 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8948124ca00c4514a4e9929615f579e9: Sending DeleteTablet for 3 replicas of tablet 48dacca0481d45e99ea3c3c66487fa99
I20260928 02:38:11.323748 18308 tablet_service.cc:1514] Processing DeleteTablet for tablet 48dacca0481d45e99ea3c3c66487fa99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:11 UTC) from {username='slave'} at 127.0.0.1:51378
I20260928 02:38:11.323921 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 17984
I20260928 02:38:11.323937 18177 tablet_service.cc:1514] Processing DeleteTablet for tablet 48dacca0481d45e99ea3c3c66487fa99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:11 UTC) from {username='slave'} at 127.0.0.1:43836
I20260928 02:38:11.324782 18410 tablet_replica.cc:330] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7: stopping tablet replica
I20260928 02:38:11.324878 18410 raft_consensus.cc:2229] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:38:11.324961 18410 raft_consensus.cc:2258] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:38:11.325354 18410 ts_tablet_manager.cc:1905] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:38:11.326829 18410 ts_tablet_manager.cc:1918] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:38:11.326893 18410 log.cc:1196] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790562862804661-392-0/minicluster-data/ts-2/wal/wals/48dacca0481d45e99ea3c3c66487fa99
I20260928 02:38:11.327119 18410 ts_tablet_manager.cc:1939] T 48dacca0481d45e99ea3c3c66487fa99 P fd9b613ccb15437daa7d40397e9ab3a7: Deleting consensus metadata
I20260928 02:38:11.327472 17909 catalog_manager.cc:4841] TS fd9b613ccb15437daa7d40397e9ab3a7 (127.0.98.3:46845): tablet 48dacca0481d45e99ea3c3c66487fa99 (table test-table [id=128eb2ff4e1f4a43af59b3e15156e272]) successfully deleted
I20260928 02:38:11.329372 18411 tablet_replica.cc:330] T 48dacca0481d45e99ea3c3c66487fa99 P ae3bd0bc51b04088ad8cf4872276d9a5: stopping tablet replica
I20260928 02:38:11.329469 18411 raft_consensus.cc:2229] T 48dacca0481d45e99ea3c3c66487fa99 P ae3bd0bc51b04088ad8cf4872276d9a5 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260928 02:38:11.329483 17908 connection.cc:535] client connection to 127.0.98.1:37117 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 02:38:11.329546 17908 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:38:11.329576 18411 raft_consensus.cc:2258] T 48dacca0481d45e99ea3c3c66487fa99 P ae3bd0bc51b04088ad8cf4872276d9a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:38:11.329947 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:11.329834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37117 (local address 127.0.0.1:55260)
0928 02:38:11.329880 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:11.329915 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 02:38:11.329957 18411 ts_tablet_manager.cc:1905] T 48dacca0481d45e99ea3c3c66487fa99 P ae3bd0bc51b04088ad8cf4872276d9a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:38:11.330018 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 18115
W20260928 02:38:11.330152 17908 catalog_manager.cc:4584] TS 2f718b17cefd4dbfb129aa3de77d4510 (127.0.98.1:37117): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 48dacca0481d45e99ea3c3c66487fa99: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37117: connect: Connection refused (error 111)
I20260928 02:38:11.334607 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:11.334504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36675 (local address 127.0.0.1:43846)
0928 02:38:11.334553 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:11.334580 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 02:38:11.334623 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 18246
I20260928 02:38:11.338775 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 17893
2026-09-28T02:38:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (694 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:38:11 dist-test-slave-fz47 krb5kdc[18415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:38:11 dist-test-slave-fz47 krb5kdc[18415](info): set up 2 sockets
Sep 28 02:38:11 dist-test-slave-fz47 krb5kdc[18415](info): commencing operation
krb5kdc: starting...
W20260928 02:38:12.276723 392 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:38:12 dist-test-slave-fz47 krb5kdc[18415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:38:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:38:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 02:38:12.381594 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41091
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:12.458734 18428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:12.458925 18428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:12.458952 18428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:12.460068 18428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:12.460111 18428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:12.460130 18428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:12.460146 18428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:12.460162 18428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:12.461632 18428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:12.461861 18428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:12.462054 18428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:12.466488 18433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:12.466464 18436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:12.466473 18434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:12.466645 18428 server_base.cc:1034] running on GCE node
I20260928 02:38:12.466957 18428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:12.467219 18428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:12.468357 18428 hybrid_clock.cc:648] HybridClock initialized: now 1790563092468346 us; error 29 us; skew 500 ppm
Sep 28 02:38:12 dist-test-slave-fz47 krb5kdc[18415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:12.470716 18428 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:38:12.471097 18428 webserver.cc:460] Webserver started at http://127.0.98.62:45429/ using document root <none> and password file <none>
I20260928 02:38:12.471297 18428 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:12.471342 18428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:12.471458 18428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:12.472446 18428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "171fca095934480396fd2da15768b239"
format_stamp: "Formatted at 2026-09-28 02:38:12 on dist-test-slave-fz47"
server_key: "cacd8ef5f812436a8d5c12e27c78323e"
server_key_iv: "95ee4db9dd7ef993abc62fb891bd63d9"
server_key_version: "encryptionkey@0"
I20260928 02:38:12.472765 18428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "171fca095934480396fd2da15768b239"
format_stamp: "Formatted at 2026-09-28 02:38:12 on dist-test-slave-fz47"
server_key: "cacd8ef5f812436a8d5c12e27c78323e"
server_key_iv: "95ee4db9dd7ef993abc62fb891bd63d9"
server_key_version: "encryptionkey@0"
I20260928 02:38:12.473973 18428 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 02:38:12.474587 18443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:12.474721 18428 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:12.474792 18428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "171fca095934480396fd2da15768b239"
format_stamp: "Formatted at 2026-09-28 02:38:12 on dist-test-slave-fz47"
server_key: "cacd8ef5f812436a8d5c12e27c78323e"
server_key_iv: "95ee4db9dd7ef993abc62fb891bd63d9"
server_key_version: "encryptionkey@0"
I20260928 02:38:12.474856 18428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:12.498719 18428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:12.500739 18428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:12.500869 18428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:12.504459 18428 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41091
I20260928 02:38:12.504499 18495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41091 every 8 connection(s)
I20260928 02:38:12.504817 18428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:38:12.505311 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 18428
I20260928 02:38:12.505434 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:38:12.505647 392 external_mini_cluster.cc:1428] Setting key e0e7a4dfd2386940a77638c856521814
I20260928 02:38:12.505802 18496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:12.508960 18496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 171fca095934480396fd2da15768b239: Bootstrap starting.
I20260928 02:38:12.509652 18496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 171fca095934480396fd2da15768b239: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:12.509931 18496 log.cc:824] T 00000000000000000000000000000000 P 171fca095934480396fd2da15768b239: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:12.510507 18496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 171fca095934480396fd2da15768b239: No bootstrap required, opened a new log
Sep 28 02:38:12 dist-test-slave-fz47 krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:38:12.512710 18496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 171fca095934480396fd2da15768b239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "171fca095934480396fd2da15768b239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41091 } }
I20260928 02:38:12.512882 18496 raft_consensus.cc:374] T 00000000000000000000000000000000 P 171fca095934480396fd2da15768b239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:12.512931 18496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 171fca095934480396fd2da15768b239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171fca095934480396fd2da15768b239, State: Initialized, Role: FOLLOWER
I20260928 02:38:12.513065 18496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 171fca095934480396fd2da15768b239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "171fca095934480396fd2da15768b239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41091 } }
I20260928 02:38:12.513145 18496 raft_consensus.cc:388] T 00000000000000000000000000000000 P 171fca095934480396fd2da15768b239 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:38:12.513177 18496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 171fca095934480396fd2da15768b239 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:38:12.513226 18496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 171fca095934480396fd2da15768b239 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:12.513891 18496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 171fca095934480396fd2da15768b239 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "171fca095934480396fd2da15768b239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41091 } }
I20260928 02:38:12.513993 18496 leader_election.cc:302] T 00000000000000000000000000000000 P 171fca095934480396fd2da15768b239 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 171fca095934480396fd2da15768b239; no voters:
I20260928 02:38:12.514140 18496 leader_election.cc:288] T 00000000000000000000000000000000 P 171fca095934480396fd2da15768b239 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:38:12.514178 18501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 171fca095934480396fd2da15768b239 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:12.514319 18501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 171fca095934480396fd2da15768b239 [term 1 LEADER]: Becoming Leader. State: Replica: 171fca095934480396fd2da15768b239, State: Running, Role: LEADER
I20260928 02:38:12.514379 18496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 171fca095934480396fd2da15768b239 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:38:12.514447 18501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 171fca095934480396fd2da15768b239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "171fca095934480396fd2da15768b239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41091 } }
I20260928 02:38:12.514761 18501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 171fca095934480396fd2da15768b239 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 171fca095934480396fd2da15768b239. Latest consensus state: current_term: 1 leader_uuid: "171fca095934480396fd2da15768b239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "171fca095934480396fd2da15768b239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41091 } } }
I20260928 02:38:12.514979 18501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 171fca095934480396fd2da15768b239 [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:12.514919 18503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 171fca095934480396fd2da15768b239 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "171fca095934480396fd2da15768b239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "171fca095934480396fd2da15768b239" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41091 } } }
I20260928 02:38:12.515352 18503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 171fca095934480396fd2da15768b239 [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:12.515484 18510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:38:12.516116 18510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:38:12.516269 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:12.506121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56514 (local address 127.0.98.62:41091)
0928 02:38:12.506318 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:12.506326 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:12.506355 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:38:12.506975 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:12.506982 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:12.507031 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:12.507141 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:12.507854 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:12.508759 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:12.509767 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:12.510216 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:12.510316 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:12.512641 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:12.512645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:12.512646 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:12.512658 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:12.514492 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:12.514894 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:12.514897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:12.514897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:12.515004 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:12.515228 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:12.515229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:12.515229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:12.515564 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:12.515776 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:12.515880 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 02:38:12.516018 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":85,"threads_started":1}
I20260928 02:38:12.517719 18510 catalog_manager.cc:1347] Generated new cluster ID: 766151cbece246009265d468d35fb9a3
I20260928 02:38:12.517774 18510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:38:12.538455 18510 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:38:12.539202 18510 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:38:12.542726 18510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 171fca095934480396fd2da15768b239: Generated new TSK 0
I20260928 02:38:12.542891 18510 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 02:38:12.570330 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41091
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:12.647822 18522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:12.648015 18522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:12.648036 18522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:12.649168 18522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:12.649207 18522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:12.649235 18522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:38:12.650746 18522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:12.650960 18522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:12.651163 18522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:12.655726 18530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:12.655861 18522 server_base.cc:1034] running on GCE node
W20260928 02:38:12.655711 18527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:12.655714 18528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:12.656208 18522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:12.656453 18522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:12.657584 18522 hybrid_clock.cc:648] HybridClock initialized: now 1790563092657570 us; error 28 us; skew 500 ppm
Sep 28 02:38:12 dist-test-slave-fz47 krb5kdc[18415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:12.659853 18522 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 02:38:12.660228 18522 webserver.cc:460] Webserver started at http://127.0.98.1:45971/ using document root <none> and password file <none>
I20260928 02:38:12.660405 18522 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:12.660450 18522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:12.660576 18522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:12.661602 18522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-0/data/instance:
uuid: "fb0c1a749498497dadefacd05ad1c527"
format_stamp: "Formatted at 2026-09-28 02:38:12 on dist-test-slave-fz47"
server_key: "ec6e447c965b1151c8c3b83eba63b78a"
server_key_iv: "0a024ebdf8178db3e122fa8c42d1abbc"
server_key_version: "encryptionkey@0"
I20260928 02:38:12.661919 18522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fb0c1a749498497dadefacd05ad1c527"
format_stamp: "Formatted at 2026-09-28 02:38:12 on dist-test-slave-fz47"
server_key: "ec6e447c965b1151c8c3b83eba63b78a"
server_key_iv: "0a024ebdf8178db3e122fa8c42d1abbc"
server_key_version: "encryptionkey@0"
I20260928 02:38:12.663074 18522 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:38:12.663733 18537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:12.663885 18522 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:12.663964 18522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "fb0c1a749498497dadefacd05ad1c527"
format_stamp: "Formatted at 2026-09-28 02:38:12 on dist-test-slave-fz47"
server_key: "ec6e447c965b1151c8c3b83eba63b78a"
server_key_iv: "0a024ebdf8178db3e122fa8c42d1abbc"
server_key_version: "encryptionkey@0"
I20260928 02:38:12.664021 18522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:12.671103 18522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:12.672724 18522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:12.672844 18522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:12.673038 18522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:38:12.673329 18522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:38:12.673369 18522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:12.673395 18522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:38:12.673418 18522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:12.678771 18522 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33769
I20260928 02:38:12.678833 18650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33769 every 8 connection(s)
I20260928 02:38:12.679085 18522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
I20260928 02:38:12.684510 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 18522
I20260928 02:38:12.684604 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-0/wal/instance
Sep 28 02:38:12 dist-test-slave-fz47 krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:38:12.684777 392 external_mini_cluster.cc:1428] Setting key c6446e56bc713b7be2e9921490499da0
I20260928 02:38:12.687731 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:12.679680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44613 (local address 127.0.98.62:41091)
0928 02:38:12.679725 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:12.679727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:12.679924 (+ 197us) server_negotiation.cc:407] Connection header received
0928 02:38:12.680791 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:12.680793 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:12.680802 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:12.680851 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:12.681444 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:12.681933 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:12.682835 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:12.683052 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:12.683097 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:12.685179 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:12.685181 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:12.685181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:12.685189 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:12.686542 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:12.686890 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:12.686890 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:12.686891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:12.686952 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:12.687140 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:12.687141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:12.687141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:12.687285 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:12.687344 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:12.687561 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 02:38:12.687641 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:38:12.687780 18653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:12.679662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41091 (local address 127.0.98.1:44613)
0928 02:38:12.679808 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:12.679841 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:12.680705 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:12.680904 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:12.680907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:12.681136 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:12.681370 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:12.681376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:12.681996 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:12.681997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:12.682775 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:12.682777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:12.683105 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:12.683106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:12.683203 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:12.683753 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:12.683760 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:12.685089 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:12.686615 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:12.686621 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:12.686623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:12.686828 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:12.687005 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:12.687006 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:12.687006 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:12.687090 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:12.687342 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:12.687345 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:12.687448 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 02:38:12.687521 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 02:38:12.687629 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
I20260928 02:38:12.688171 18651 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41091
I20260928 02:38:12.688246 18651 heartbeater.cc:464] Registering TS with master...
I20260928 02:38:12.688400 18651 heartbeater.cc:511] Master 127.0.98.62:41091 requested a full tablet report, sending...
I20260928 02:38:12.688946 18460 ts_manager.cc:194] Registered new tserver with Master: fb0c1a749498497dadefacd05ad1c527 (127.0.98.1:33769)
I20260928 02:38:12.689855 18460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44613
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 02:38:12.705556 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41091
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:12.782392 18656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:12.782570 18656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:12.782591 18656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:12.783715 18656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:12.783751 18656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:12.783775 18656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:38:12.785161 18656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:12.785359 18656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:12.785530 18656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:12.789958 18662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:12.789983 18656 server_base.cc:1034] running on GCE node
W20260928 02:38:12.789954 18661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:12.789954 18664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:12.790305 18656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:12.790496 18656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:12.791625 18656 hybrid_clock.cc:648] HybridClock initialized: now 1790563092791613 us; error 30 us; skew 500 ppm
Sep 28 02:38:12 dist-test-slave-fz47 krb5kdc[18415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:12.793921 18656 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 02:38:12.794332 18656 webserver.cc:460] Webserver started at http://127.0.98.2:38713/ using document root <none> and password file <none>
I20260928 02:38:12.794541 18656 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:12.794584 18656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:12.794685 18656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:12.795641 18656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-1/data/instance:
uuid: "cc1cf9b3459e4bb3993058442eb3b20d"
format_stamp: "Formatted at 2026-09-28 02:38:12 on dist-test-slave-fz47"
server_key: "0b494b1fae09a2a94cea56513e924378"
server_key_iv: "3d31f887f449f501195e9a961ab49b6c"
server_key_version: "encryptionkey@0"
I20260928 02:38:12.795943 18656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cc1cf9b3459e4bb3993058442eb3b20d"
format_stamp: "Formatted at 2026-09-28 02:38:12 on dist-test-slave-fz47"
server_key: "0b494b1fae09a2a94cea56513e924378"
server_key_iv: "3d31f887f449f501195e9a961ab49b6c"
server_key_version: "encryptionkey@0"
I20260928 02:38:12.797098 18656 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:38:12.797788 18671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:12.797936 18656 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:12.798008 18656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "cc1cf9b3459e4bb3993058442eb3b20d"
format_stamp: "Formatted at 2026-09-28 02:38:12 on dist-test-slave-fz47"
server_key: "0b494b1fae09a2a94cea56513e924378"
server_key_iv: "3d31f887f449f501195e9a961ab49b6c"
server_key_version: "encryptionkey@0"
I20260928 02:38:12.798060 18656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:12.815274 18656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:12.817018 18656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:12.817139 18656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:12.817355 18656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:38:12.817638 18656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:38:12.817670 18656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:12.817691 18656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:38:12.817713 18656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:12.822721 18656 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45589
I20260928 02:38:12.822791 18784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45589 every 8 connection(s)
I20260928 02:38:12.823032 18656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 02:38:12 dist-test-slave-fz47 krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:38:12.829394 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 18656
I20260928 02:38:12.829517 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-1/wal/instance
I20260928 02:38:12.829696 392 external_mini_cluster.cc:1428] Setting key 216361358423888366c07c7b14b86952
I20260928 02:38:12.831686 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:12.823645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43315 (local address 127.0.98.62:41091)
0928 02:38:12.823706 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:12.823708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:12.823890 (+ 182us) server_negotiation.cc:407] Connection header received
0928 02:38:12.824790 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:12.824792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:12.824804 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:12.824850 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:12.825471 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:12.825963 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:12.826916 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:12.827118 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:12.827172 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:12.829048 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:12.829049 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:12.829050 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:12.829056 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:12.830421 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:12.830832 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:12.830833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:12.830833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:12.830881 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:12.831100 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:12.831101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:12.831101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:12.831191 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:12.831233 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:12.831503 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 02:38:12.831577 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 02:38:12.831868 18787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:12.823615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41091 (local address 127.0.98.2:43315)
0928 02:38:12.823769 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:12.823803 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:12.824682 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:12.824926 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:12.824929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:12.825179 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:12.825380 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:12.825384 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:12.826047 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:12.826048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:12.826837 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:12.826838 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:12.827183 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:12.827184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:12.827290 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:12.827723 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:12.827730 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:12.828956 (+ 1226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:12.830491 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:12.830498 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:12.830500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:12.830765 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:12.830931 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:12.830932 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:12.830932 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:12.831049 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:12.831238 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:12.831242 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:12.831386 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 02:38:12.831484 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 02:38:12.831652 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20260928 02:38:12.832258 18785 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41091
I20260928 02:38:12.832337 18785 heartbeater.cc:464] Registering TS with master...
I20260928 02:38:12.832492 18785 heartbeater.cc:511] Master 127.0.98.62:41091 requested a full tablet report, sending...
I20260928 02:38:12.832834 18460 ts_manager.cc:194] Registered new tserver with Master: cc1cf9b3459e4bb3993058442eb3b20d (127.0.98.2:45589)
I20260928 02:38:12.833313 18460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43315
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 02:38:12.851136 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41091
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:12.927968 18790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:12.928164 18790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:12.928186 18790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:12.929322 18790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:12.929361 18790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:12.929390 18790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:38:12.930826 18790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:12.931042 18790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:12.931232 18790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:12.935874 18796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:12.935876 18795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:12.935906 18790 server_base.cc:1034] running on GCE node
W20260928 02:38:12.935876 18798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:12.936260 18790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:12.936480 18790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:12.937611 18790 hybrid_clock.cc:648] HybridClock initialized: now 1790563092937596 us; error 33 us; skew 500 ppm
Sep 28 02:38:12 dist-test-slave-fz47 krb5kdc[18415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:12.939815 18790 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 02:38:12.940174 18790 webserver.cc:460] Webserver started at http://127.0.98.3:46149/ using document root <none> and password file <none>
I20260928 02:38:12.940344 18790 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:12.940380 18790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:12.940467 18790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:12.941416 18790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-2/data/instance:
uuid: "e2e476d60f88484ca97346ef0ca2a1f8"
format_stamp: "Formatted at 2026-09-28 02:38:12 on dist-test-slave-fz47"
server_key: "98a3905b51b595b8d7e3449064d9fc03"
server_key_iv: "583b349a16c22487981b1e7e6515b35b"
server_key_version: "encryptionkey@0"
I20260928 02:38:12.941709 18790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e2e476d60f88484ca97346ef0ca2a1f8"
format_stamp: "Formatted at 2026-09-28 02:38:12 on dist-test-slave-fz47"
server_key: "98a3905b51b595b8d7e3449064d9fc03"
server_key_iv: "583b349a16c22487981b1e7e6515b35b"
server_key_version: "encryptionkey@0"
I20260928 02:38:12.942839 18790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:38:12.943426 18805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:12.943573 18790 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:12.943648 18790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "e2e476d60f88484ca97346ef0ca2a1f8"
format_stamp: "Formatted at 2026-09-28 02:38:12 on dist-test-slave-fz47"
server_key: "98a3905b51b595b8d7e3449064d9fc03"
server_key_iv: "583b349a16c22487981b1e7e6515b35b"
server_key_version: "encryptionkey@0"
I20260928 02:38:12.943699 18790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:12.957916 18790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:12.960089 18790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:12.960201 18790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:12.960377 18790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:38:12.960630 18790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:38:12.960659 18790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:12.960680 18790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:38:12.960695 18790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:12.965927 18790 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44147
I20260928 02:38:12.965987 18918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44147 every 8 connection(s)
I20260928 02:38:12.966260 18790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 02:38:12 dist-test-slave-fz47 krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:38:12.974602 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:12.966899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46855 (local address 127.0.98.62:41091)
0928 02:38:12.966976 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:12.966978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:12.967132 (+ 154us) server_negotiation.cc:407] Connection header received
0928 02:38:12.968040 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:12.968041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:12.968052 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:12.968100 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:12.968645 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:12.969126 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:12.970165 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:12.970344 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:12.970385 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:12.972271 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:12.972272 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:12.972273 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:12.972279 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:12.973530 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:12.973858 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:12.973858 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:12.973859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:12.973904 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:12.974147 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:12.974148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:12.974148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:12.974237 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:12.974281 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:12.974484 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 02:38:12.974537 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:38:12.974828 18921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:12.966882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41091 (local address 127.0.98.3:46855)
0928 02:38:12.967017 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:12.967052 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:12.967954 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:12.968156 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:12.968159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:12.968390 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:12.968582 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:12.968586 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:12.969197 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:12.969197 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:12.970099 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:12.970102 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:12.970395 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:12.970395 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:12.970526 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:12.970924 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:12.970932 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:12.972185 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:12.973590 (+ 1405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:12.973595 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:12.973597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:12.973797 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:12.973984 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:12.973985 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:12.973985 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:12.974091 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:12.974282 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:12.974285 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:12.974388 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 02:38:12.974468 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:38:12.974606 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":76,"threads_started":1}
I20260928 02:38:12.975049 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 18790
I20260928 02:38:12.975152 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-2/wal/instance
I20260928 02:38:12.975333 18919 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41091
I20260928 02:38:12.975348 392 external_mini_cluster.cc:1428] Setting key b289ba717b9fbf92fdc96eba4ef3d629
I20260928 02:38:12.975418 18919 heartbeater.cc:464] Registering TS with master...
I20260928 02:38:12.975622 18919 heartbeater.cc:511] Master 127.0.98.62:41091 requested a full tablet report, sending...
I20260928 02:38:12.975972 18459 ts_manager.cc:194] Registered new tserver with Master: e2e476d60f88484ca97346ef0ca2a1f8 (127.0.98.3:44147)
I20260928 02:38:12.976346 18459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46855
I20260928 02:38:12.986768 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:38:12.993501 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:12.988011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56524 (local address 127.0.98.62:41091)
0928 02:38:12.988064 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:12.988065 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:12.988171 (+ 106us) server_negotiation.cc:407] Connection header received
0928 02:38:12.988210 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:12.988211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:12.988219 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:12.988260 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:12.988733 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:12.989192 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:12.989899 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:12.990077 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:12.990118 (+ 41us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:12.991292 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:12.991293 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:12.991293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:12.991300 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:12.992453 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:12.992783 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:12.992784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:12.992784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:12.992830 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:12.992974 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:12.992974 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:12.992974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:12.993081 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:12.993159 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:12.993336 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 02:38:12.993413 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 02:38:12.995047 18459 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:38:12.996541 18459 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:38:13.002737 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:12.998490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33769 (local address 127.0.0.1:48762)
0928 02:38:12.998758 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:12.998796 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:12.998861 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:12.999519 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:12.999520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:12.999522 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:12.999555 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:12.999557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.000463 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.000464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:13.001882 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:13.001883 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.002495 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.002496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:13.002567 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:13.002570 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:38:13.002612 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 02:38:13.002653 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":73,"thread_start_us":74,"threads_started":1}
I20260928 02:38:13.002743 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:12.998826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48762 (local address 127.0.98.1:33769)
0928 02:38:12.999308 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:12.999312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:12.999321 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:38:12.999356 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:12.999358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:12.999372 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:12.999438 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:12.999624 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.000362 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.001948 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.002421 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.002482 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:13.002535 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:13.002635 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 02:38:13.002669 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":74,"threads_started":1}
I20260928 02:38:13.002811 18930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:12.998636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38460 (local address 127.0.98.2:45589)
0928 02:38:12.998813 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:12.998817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:12.998954 (+ 137us) server_negotiation.cc:407] Connection header received
0928 02:38:12.998993 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:12.998995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:12.999007 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:12.999060 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:12.999229 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.000321 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.001568 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.002242 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.002309 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:13.002365 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:13.002697 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 02:38:13.002737 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":68,"threads_started":1}
I20260928 02:38:13.002988 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:12.998639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49814 (local address 127.0.98.3:44147)
0928 02:38:12.999016 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:12.999020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:12.999027 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:38:12.999066 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:12.999068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:12.999084 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:12.999165 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:12.999297 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.000358 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.001856 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.002513 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.002568 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:13.002615 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:13.002894 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 02:38:13.002933 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":85,"threads_started":1}
I20260928 02:38:13.002839 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:12.998491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44147 (local address 127.0.0.1:49814)
0928 02:38:12.998815 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:12.998824 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:12.998877 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:12.999171 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:12.999172 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:12.999174 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:12.999213 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:12.999215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.000483 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.000483 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:13.001772 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:13.001774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.002648 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.002649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:13.002704 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:13.002708 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:38:13.002751 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 02:38:13.002785 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":84,"thread_start_us":49,"threads_started":1}
I20260928 02:38:13.003206 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:12.998490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45589 (local address 127.0.0.1:38460)
0928 02:38:12.998858 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:12.998866 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:12.998907 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:12.999090 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:12.999092 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:12.999094 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:12.999164 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:12.999165 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.000453 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.000454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:13.001508 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:13.001511 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.002397 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.002398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:13.002473 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:13.002485 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 02:38:13.002552 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 02:38:13.002603 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":159,"thread_start_us":79,"threads_started":1}
I20260928 02:38:13.003832 18585 tablet_service.cc:1467] Processing CreateTablet for tablet bd03b0475f5b4146a47333ca041325e4 (DEFAULT_TABLE table=test-table [id=11cebba647ff4daf9fd88e6f135f2d2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:13.003830 18853 tablet_service.cc:1467] Processing CreateTablet for tablet bd03b0475f5b4146a47333ca041325e4 (DEFAULT_TABLE table=test-table [id=11cebba647ff4daf9fd88e6f135f2d2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:13.004145 18853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd03b0475f5b4146a47333ca041325e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:13.004145 18585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd03b0475f5b4146a47333ca041325e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:13.007285 18933 tablet_bootstrap.cc:492] T bd03b0475f5b4146a47333ca041325e4 P e2e476d60f88484ca97346ef0ca2a1f8: Bootstrap starting.
I20260928 02:38:13.007653 18719 tablet_service.cc:1467] Processing CreateTablet for tablet bd03b0475f5b4146a47333ca041325e4 (DEFAULT_TABLE table=test-table [id=11cebba647ff4daf9fd88e6f135f2d2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:13.007855 18933 tablet_bootstrap.cc:654] T bd03b0475f5b4146a47333ca041325e4 P e2e476d60f88484ca97346ef0ca2a1f8: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:13.007882 18719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd03b0475f5b4146a47333ca041325e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:13.008395 18933 log.cc:824] T bd03b0475f5b4146a47333ca041325e4 P e2e476d60f88484ca97346ef0ca2a1f8: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:13.008674 18934 tablet_bootstrap.cc:492] T bd03b0475f5b4146a47333ca041325e4 P fb0c1a749498497dadefacd05ad1c527: Bootstrap starting.
I20260928 02:38:13.009157 18933 tablet_bootstrap.cc:492] T bd03b0475f5b4146a47333ca041325e4 P e2e476d60f88484ca97346ef0ca2a1f8: No bootstrap required, opened a new log
I20260928 02:38:13.009217 18934 tablet_bootstrap.cc:654] T bd03b0475f5b4146a47333ca041325e4 P fb0c1a749498497dadefacd05ad1c527: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:13.009217 18933 ts_tablet_manager.cc:1397] T bd03b0475f5b4146a47333ca041325e4 P e2e476d60f88484ca97346ef0ca2a1f8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:38:13.009752 18934 log.cc:824] T bd03b0475f5b4146a47333ca041325e4 P fb0c1a749498497dadefacd05ad1c527: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:13.010488 18934 tablet_bootstrap.cc:492] T bd03b0475f5b4146a47333ca041325e4 P fb0c1a749498497dadefacd05ad1c527: No bootstrap required, opened a new log
I20260928 02:38:13.010545 18934 ts_tablet_manager.cc:1397] T bd03b0475f5b4146a47333ca041325e4 P fb0c1a749498497dadefacd05ad1c527: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:38:13.010819 18937 tablet_bootstrap.cc:492] T bd03b0475f5b4146a47333ca041325e4 P cc1cf9b3459e4bb3993058442eb3b20d: Bootstrap starting.
I20260928 02:38:13.011101 18933 raft_consensus.cc:348] T bd03b0475f5b4146a47333ca041325e4 P e2e476d60f88484ca97346ef0ca2a1f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e476d60f88484ca97346ef0ca2a1f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44147 } } peers { permanent_uuid: "fb0c1a749498497dadefacd05ad1c527" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "cc1cf9b3459e4bb3993058442eb3b20d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45589 } }
I20260928 02:38:13.011286 18937 tablet_bootstrap.cc:654] T bd03b0475f5b4146a47333ca041325e4 P cc1cf9b3459e4bb3993058442eb3b20d: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:13.011294 18933 raft_consensus.cc:374] T bd03b0475f5b4146a47333ca041325e4 P e2e476d60f88484ca97346ef0ca2a1f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:13.011332 18933 raft_consensus.cc:729] T bd03b0475f5b4146a47333ca041325e4 P e2e476d60f88484ca97346ef0ca2a1f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2e476d60f88484ca97346ef0ca2a1f8, State: Initialized, Role: FOLLOWER
I20260928 02:38:13.011471 18933 consensus_queue.cc:260] T bd03b0475f5b4146a47333ca041325e4 P e2e476d60f88484ca97346ef0ca2a1f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e476d60f88484ca97346ef0ca2a1f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44147 } } peers { permanent_uuid: "fb0c1a749498497dadefacd05ad1c527" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "cc1cf9b3459e4bb3993058442eb3b20d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45589 } }
I20260928 02:38:13.011755 18933 ts_tablet_manager.cc:1428] T bd03b0475f5b4146a47333ca041325e4 P e2e476d60f88484ca97346ef0ca2a1f8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:38:13.011766 18937 log.cc:824] T bd03b0475f5b4146a47333ca041325e4 P cc1cf9b3459e4bb3993058442eb3b20d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:13.011868 18919 heartbeater.cc:503] Master 127.0.98.62:41091 was elected leader, sending a full tablet report...
I20260928 02:38:13.012442 18937 tablet_bootstrap.cc:492] T bd03b0475f5b4146a47333ca041325e4 P cc1cf9b3459e4bb3993058442eb3b20d: No bootstrap required, opened a new log
I20260928 02:38:13.012509 18937 ts_tablet_manager.cc:1397] T bd03b0475f5b4146a47333ca041325e4 P cc1cf9b3459e4bb3993058442eb3b20d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:38:13.012408 18934 raft_consensus.cc:348] T bd03b0475f5b4146a47333ca041325e4 P fb0c1a749498497dadefacd05ad1c527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e476d60f88484ca97346ef0ca2a1f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44147 } } peers { permanent_uuid: "fb0c1a749498497dadefacd05ad1c527" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "cc1cf9b3459e4bb3993058442eb3b20d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45589 } }
I20260928 02:38:13.012557 18934 raft_consensus.cc:374] T bd03b0475f5b4146a47333ca041325e4 P fb0c1a749498497dadefacd05ad1c527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:13.012598 18934 raft_consensus.cc:729] T bd03b0475f5b4146a47333ca041325e4 P fb0c1a749498497dadefacd05ad1c527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb0c1a749498497dadefacd05ad1c527, State: Initialized, Role: FOLLOWER
I20260928 02:38:13.012738 18934 consensus_queue.cc:260] T bd03b0475f5b4146a47333ca041325e4 P fb0c1a749498497dadefacd05ad1c527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e476d60f88484ca97346ef0ca2a1f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44147 } } peers { permanent_uuid: "fb0c1a749498497dadefacd05ad1c527" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "cc1cf9b3459e4bb3993058442eb3b20d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45589 } }
I20260928 02:38:13.012998 18934 ts_tablet_manager.cc:1428] T bd03b0475f5b4146a47333ca041325e4 P fb0c1a749498497dadefacd05ad1c527: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:38:13.013054 18651 heartbeater.cc:503] Master 127.0.98.62:41091 was elected leader, sending a full tablet report...
I20260928 02:38:13.014051 18937 raft_consensus.cc:348] T bd03b0475f5b4146a47333ca041325e4 P cc1cf9b3459e4bb3993058442eb3b20d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e476d60f88484ca97346ef0ca2a1f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44147 } } peers { permanent_uuid: "fb0c1a749498497dadefacd05ad1c527" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "cc1cf9b3459e4bb3993058442eb3b20d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45589 } }
I20260928 02:38:13.014183 18937 raft_consensus.cc:374] T bd03b0475f5b4146a47333ca041325e4 P cc1cf9b3459e4bb3993058442eb3b20d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:13.014204 18937 raft_consensus.cc:729] T bd03b0475f5b4146a47333ca041325e4 P cc1cf9b3459e4bb3993058442eb3b20d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc1cf9b3459e4bb3993058442eb3b20d, State: Initialized, Role: FOLLOWER
I20260928 02:38:13.014317 18937 consensus_queue.cc:260] T bd03b0475f5b4146a47333ca041325e4 P cc1cf9b3459e4bb3993058442eb3b20d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e476d60f88484ca97346ef0ca2a1f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44147 } } peers { permanent_uuid: "fb0c1a749498497dadefacd05ad1c527" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "cc1cf9b3459e4bb3993058442eb3b20d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45589 } }
I20260928 02:38:13.014541 18937 ts_tablet_manager.cc:1428] T bd03b0475f5b4146a47333ca041325e4 P cc1cf9b3459e4bb3993058442eb3b20d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:38:13.014581 18785 heartbeater.cc:503] Master 127.0.98.62:41091 was elected leader, sending a full tablet report...
W20260928 02:38:13.073525 18786 tablet.cc:2369] T bd03b0475f5b4146a47333ca041325e4 P cc1cf9b3459e4bb3993058442eb3b20d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:38:13.179770 18652 tablet.cc:2369] T bd03b0475f5b4146a47333ca041325e4 P fb0c1a749498497dadefacd05ad1c527: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:38:13.212477 18940 raft_consensus.cc:482] T bd03b0475f5b4146a47333ca041325e4 P fb0c1a749498497dadefacd05ad1c527 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:38:13.212610 18940 raft_consensus.cc:504] T bd03b0475f5b4146a47333ca041325e4 P fb0c1a749498497dadefacd05ad1c527 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e476d60f88484ca97346ef0ca2a1f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44147 } } peers { permanent_uuid: "fb0c1a749498497dadefacd05ad1c527" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "cc1cf9b3459e4bb3993058442eb3b20d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45589 } }
I20260928 02:38:13.212967 18940 leader_election.cc:288] T bd03b0475f5b4146a47333ca041325e4 P fb0c1a749498497dadefacd05ad1c527 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2e476d60f88484ca97346ef0ca2a1f8 (127.0.98.3:44147), cc1cf9b3459e4bb3993058442eb3b20d (127.0.98.2:45589)
I20260928 02:38:13.215927 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:13.213235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51835 (local address 127.0.98.3:44147)
0928 02:38:13.213303 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:13.213305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:13.213372 (+ 67us) server_negotiation.cc:407] Connection header received
0928 02:38:13.213429 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:13.213430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:13.213439 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:13.213483 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:13.213635 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.214126 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.215173 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.215580 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.215632 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:13.215640 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:13.215791 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 02:38:13.215861 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:38:13.215935 18930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:13.213208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42339 (local address 127.0.98.2:45589)
0928 02:38:13.213281 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:13.213283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:13.213391 (+ 108us) server_negotiation.cc:407] Connection header received
0928 02:38:13.213449 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:13.213450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:13.213459 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:13.213512 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:13.213677 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.214131 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.215174 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.215608 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.215677 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:13.215688 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:13.215809 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 02:38:13.215863 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:38:13.215958 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:13.213150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45589 (local address 127.0.98.1:42339)
0928 02:38:13.213320 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:13.213329 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:13.213393 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:13.213552 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:13.213552 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:13.213554 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:13.213613 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:13.213615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.214218 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.214219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:13.215095 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:13.215098 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.215686 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.215687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:13.215746 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:13.215750 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:38:13.215798 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 02:38:13.215851 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":15,"thread_start_us":65,"threads_started":1}
I20260928 02:38:13.215945 18943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:13.213146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44147 (local address 127.0.98.1:51835)
0928 02:38:13.213303 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:13.213314 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:13.213373 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:13.213496 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:13.213497 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:13.213500 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:13.213578 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:13.213580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.214256 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.214257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:13.215096 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:13.215099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.215656 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.215657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:13.215729 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:13.215734 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:38:13.215774 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 02:38:13.215817 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":68,"threads_started":1}
I20260928 02:38:13.216326 18739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd03b0475f5b4146a47333ca041325e4" candidate_uuid: "fb0c1a749498497dadefacd05ad1c527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc1cf9b3459e4bb3993058442eb3b20d" is_pre_election: true
I20260928 02:38:13.216492 18739 raft_consensus.cc:2454] T bd03b0475f5b4146a47333ca041325e4 P cc1cf9b3459e4bb3993058442eb3b20d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb0c1a749498497dadefacd05ad1c527 in term 0.
I20260928 02:38:13.216612 18873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd03b0475f5b4146a47333ca041325e4" candidate_uuid: "fb0c1a749498497dadefacd05ad1c527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2e476d60f88484ca97346ef0ca2a1f8" is_pre_election: true
I20260928 02:38:13.216735 18538 leader_election.cc:302] T bd03b0475f5b4146a47333ca041325e4 P fb0c1a749498497dadefacd05ad1c527 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc1cf9b3459e4bb3993058442eb3b20d, fb0c1a749498497dadefacd05ad1c527; no voters:
W20260928 02:38:13.216761 18920 tablet.cc:2369] T bd03b0475f5b4146a47333ca041325e4 P e2e476d60f88484ca97346ef0ca2a1f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:38:13.216759 18873 raft_consensus.cc:2454] T bd03b0475f5b4146a47333ca041325e4 P e2e476d60f88484ca97346ef0ca2a1f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb0c1a749498497dadefacd05ad1c527 in term 0.
I20260928 02:38:13.216866 18940 raft_consensus.cc:2781] T bd03b0475f5b4146a47333ca041325e4 P fb0c1a749498497dadefacd05ad1c527 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:38:13.216913 18940 raft_consensus.cc:482] T bd03b0475f5b4146a47333ca041325e4 P fb0c1a749498497dadefacd05ad1c527 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:38:13.216933 18940 raft_consensus.cc:3037] T bd03b0475f5b4146a47333ca041325e4 P fb0c1a749498497dadefacd05ad1c527 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:13.217640 18940 raft_consensus.cc:504] T bd03b0475f5b4146a47333ca041325e4 P fb0c1a749498497dadefacd05ad1c527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e476d60f88484ca97346ef0ca2a1f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44147 } } peers { permanent_uuid: "fb0c1a749498497dadefacd05ad1c527" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "cc1cf9b3459e4bb3993058442eb3b20d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45589 } }
I20260928 02:38:13.217748 18940 leader_election.cc:288] T bd03b0475f5b4146a47333ca041325e4 P fb0c1a749498497dadefacd05ad1c527 [CANDIDATE]: Term 1 election: Requested vote from peers e2e476d60f88484ca97346ef0ca2a1f8 (127.0.98.3:44147), cc1cf9b3459e4bb3993058442eb3b20d (127.0.98.2:45589)
I20260928 02:38:13.217921 18739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd03b0475f5b4146a47333ca041325e4" candidate_uuid: "fb0c1a749498497dadefacd05ad1c527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc1cf9b3459e4bb3993058442eb3b20d"
I20260928 02:38:13.217893 18873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd03b0475f5b4146a47333ca041325e4" candidate_uuid: "fb0c1a749498497dadefacd05ad1c527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2e476d60f88484ca97346ef0ca2a1f8"
I20260928 02:38:13.217984 18873 raft_consensus.cc:3037] T bd03b0475f5b4146a47333ca041325e4 P e2e476d60f88484ca97346ef0ca2a1f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:13.217983 18739 raft_consensus.cc:3037] T bd03b0475f5b4146a47333ca041325e4 P cc1cf9b3459e4bb3993058442eb3b20d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:13.218600 18739 raft_consensus.cc:2454] T bd03b0475f5b4146a47333ca041325e4 P cc1cf9b3459e4bb3993058442eb3b20d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb0c1a749498497dadefacd05ad1c527 in term 1.
I20260928 02:38:13.218678 18873 raft_consensus.cc:2454] T bd03b0475f5b4146a47333ca041325e4 P e2e476d60f88484ca97346ef0ca2a1f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb0c1a749498497dadefacd05ad1c527 in term 1.
I20260928 02:38:13.218739 18538 leader_election.cc:302] T bd03b0475f5b4146a47333ca041325e4 P fb0c1a749498497dadefacd05ad1c527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc1cf9b3459e4bb3993058442eb3b20d, fb0c1a749498497dadefacd05ad1c527; no voters:
I20260928 02:38:13.218825 18940 raft_consensus.cc:2781] T bd03b0475f5b4146a47333ca041325e4 P fb0c1a749498497dadefacd05ad1c527 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:13.218948 18940 raft_consensus.cc:686] T bd03b0475f5b4146a47333ca041325e4 P fb0c1a749498497dadefacd05ad1c527 [term 1 LEADER]: Becoming Leader. State: Replica: fb0c1a749498497dadefacd05ad1c527, State: Running, Role: LEADER
I20260928 02:38:13.219019 18940 consensus_queue.cc:237] T bd03b0475f5b4146a47333ca041325e4 P fb0c1a749498497dadefacd05ad1c527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e476d60f88484ca97346ef0ca2a1f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44147 } } peers { permanent_uuid: "fb0c1a749498497dadefacd05ad1c527" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } } peers { permanent_uuid: "cc1cf9b3459e4bb3993058442eb3b20d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45589 } }
I20260928 02:38:13.219736 18459 catalog_manager.cc:5495] T bd03b0475f5b4146a47333ca041325e4 P fb0c1a749498497dadefacd05ad1c527 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb0c1a749498497dadefacd05ad1c527 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "fb0c1a749498497dadefacd05ad1c527" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e476d60f88484ca97346ef0ca2a1f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb0c1a749498497dadefacd05ad1c527" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc1cf9b3459e4bb3993058442eb3b20d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45589 } health_report { overall_health: UNKNOWN } } }
I20260928 02:38:13.228128 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:13.225977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48770 (local address 127.0.98.1:33769)
0928 02:38:13.226026 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:13.226028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:13.226035 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:38:13.226065 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:13.226066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:13.226084 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:13.226144 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:38:13.226257 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.226707 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.227430 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.227627 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.227677 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:13.227818 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:38:13.227911 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:38:13.227992 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:13.228048 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 02:38:13.228056 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 02:38:13.229646 18739 raft_consensus.cc:1261] T bd03b0475f5b4146a47333ca041325e4 P cc1cf9b3459e4bb3993058442eb3b20d [term 1 FOLLOWER]: Refusing update from remote peer fb0c1a749498497dadefacd05ad1c527: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:38:13.229655 18873 raft_consensus.cc:1261] T bd03b0475f5b4146a47333ca041325e4 P e2e476d60f88484ca97346ef0ca2a1f8 [term 1 FOLLOWER]: Refusing update from remote peer fb0c1a749498497dadefacd05ad1c527: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:38:13.229946 18944 consensus_queue.cc:1035] T bd03b0475f5b4146a47333ca041325e4 P fb0c1a749498497dadefacd05ad1c527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2e476d60f88484ca97346ef0ca2a1f8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:38:13.230085 18940 consensus_queue.cc:1035] T bd03b0475f5b4146a47333ca041325e4 P fb0c1a749498497dadefacd05ad1c527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc1cf9b3459e4bb3993058442eb3b20d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:38:13.232336 18949 mvcc.cc:205] Tried to move back new op lower bound from 7334146429866328064 to 7334146429825687552. Current Snapshot: MvccSnapshot[applied={T|T < 7334146429866328064}]
I20260928 02:38:13.232072 18947 mvcc.cc:205] Tried to move back new op lower bound from 7334146429866328064 to 7334146429825687552. Current Snapshot: MvccSnapshot[applied={T|T < 7334146429866328064}]
I20260928 02:38:13.234956 18459 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:38:13.235044 18459 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:38:13.235673 18459 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 171fca095934480396fd2da15768b239: Sending DeleteTablet for 3 replicas of tablet bd03b0475f5b4146a47333ca041325e4
I20260928 02:38:13.235937 18585 tablet_service.cc:1514] Processing DeleteTablet for tablet bd03b0475f5b4146a47333ca041325e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48762
I20260928 02:38:13.236006 18853 tablet_service.cc:1514] Processing DeleteTablet for tablet bd03b0475f5b4146a47333ca041325e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49814
I20260928 02:38:13.236060 18719 tablet_service.cc:1514] Processing DeleteTablet for tablet bd03b0475f5b4146a47333ca041325e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38460
I20260928 02:38:13.236193 18957 tablet_replica.cc:330] T bd03b0475f5b4146a47333ca041325e4 P e2e476d60f88484ca97346ef0ca2a1f8: stopping tablet replica
I20260928 02:38:13.236279 18957 raft_consensus.cc:2229] T bd03b0475f5b4146a47333ca041325e4 P e2e476d60f88484ca97346ef0ca2a1f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:38:13.236320 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 18522
I20260928 02:38:13.236371 18957 raft_consensus.cc:2258] T bd03b0475f5b4146a47333ca041325e4 P e2e476d60f88484ca97346ef0ca2a1f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:38:13.237812 18958 tablet_replica.cc:330] T bd03b0475f5b4146a47333ca041325e4 P cc1cf9b3459e4bb3993058442eb3b20d: stopping tablet replica
I20260928 02:38:13.237895 18958 raft_consensus.cc:2229] T bd03b0475f5b4146a47333ca041325e4 P cc1cf9b3459e4bb3993058442eb3b20d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:38:13.237979 18958 raft_consensus.cc:2258] T bd03b0475f5b4146a47333ca041325e4 P cc1cf9b3459e4bb3993058442eb3b20d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:38:13.238209 18957 ts_tablet_manager.cc:1905] T bd03b0475f5b4146a47333ca041325e4 P e2e476d60f88484ca97346ef0ca2a1f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:38:13.238325 18958 ts_tablet_manager.cc:1905] T bd03b0475f5b4146a47333ca041325e4 P cc1cf9b3459e4bb3993058442eb3b20d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:38:13.239871 18958 ts_tablet_manager.cc:1918] T bd03b0475f5b4146a47333ca041325e4 P cc1cf9b3459e4bb3993058442eb3b20d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:38:13.239871 18957 ts_tablet_manager.cc:1918] T bd03b0475f5b4146a47333ca041325e4 P e2e476d60f88484ca97346ef0ca2a1f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:38:13.239943 18958 log.cc:1196] T bd03b0475f5b4146a47333ca041325e4 P cc1cf9b3459e4bb3993058442eb3b20d: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-1/wal/wals/bd03b0475f5b4146a47333ca041325e4
I20260928 02:38:13.239941 18957 log.cc:1196] T bd03b0475f5b4146a47333ca041325e4 P e2e476d60f88484ca97346ef0ca2a1f8: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790562862804661-392-0/minicluster-data/ts-2/wal/wals/bd03b0475f5b4146a47333ca041325e4
I20260928 02:38:13.240200 18957 ts_tablet_manager.cc:1939] T bd03b0475f5b4146a47333ca041325e4 P e2e476d60f88484ca97346ef0ca2a1f8: Deleting consensus metadata
I20260928 02:38:13.240213 18958 ts_tablet_manager.cc:1939] T bd03b0475f5b4146a47333ca041325e4 P cc1cf9b3459e4bb3993058442eb3b20d: Deleting consensus metadata
I20260928 02:38:13.240550 18444 catalog_manager.cc:4841] TS cc1cf9b3459e4bb3993058442eb3b20d (127.0.98.2:45589): tablet bd03b0475f5b4146a47333ca041325e4 (table test-table [id=11cebba647ff4daf9fd88e6f135f2d2a]) successfully deleted
I20260928 02:38:13.240594 18447 catalog_manager.cc:4841] TS e2e476d60f88484ca97346ef0ca2a1f8 (127.0.98.3:44147): tablet bd03b0475f5b4146a47333ca041325e4 (table test-table [id=11cebba647ff4daf9fd88e6f135f2d2a]) successfully deleted
W20260928 02:38:13.240983 18446 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33769 (error 108)
I20260928 02:38:13.241212 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 18656
I20260928 02:38:13.241433 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:13.241333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33769 (local address 127.0.0.1:48786)
0928 02:38:13.241376 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:13.241405 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 02:38:13.241516 18446 catalog_manager.cc:4584] TS fb0c1a749498497dadefacd05ad1c527 (127.0.98.1:33769): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bd03b0475f5b4146a47333ca041325e4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33769: connect: Connection refused (error 111)
I20260928 02:38:13.245860 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 18790
I20260928 02:38:13.249686 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 18428
2026-09-28T02:38:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1911 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T02:38:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:38:13Z Disabled control of system clock
I20260928 02:38:13.274339 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39301
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:38:13.351002 18963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:13.351202 18963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:13.351219 18963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:13.352617 18963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:13.352667 18963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:13.352684 18963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:13.352699 18963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:13.352725 18963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:13.354583 18963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39301
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:13.354773 18963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:13.355001 18963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:13.359395 18968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:13.359401 18969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:13.359416 18971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:13.359484 18963 server_base.cc:1034] running on GCE node
I20260928 02:38:13.359787 18963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:13.360095 18963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:13.361229 18963 hybrid_clock.cc:648] HybridClock initialized: now 1790563093361216 us; error 29 us; skew 500 ppm
I20260928 02:38:13.362355 18963 webserver.cc:460] Webserver started at http://127.0.98.62:34615/ using document root <none> and password file <none>
I20260928 02:38:13.362576 18963 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:13.362624 18963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:13.362747 18963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:13.363947 18963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "7bd630221fd745ac850a95e04d3e21b8"
format_stamp: "Formatted at 2026-09-28 02:38:13 on dist-test-slave-fz47"
server_key: "72827aeb51628d9b1d46a9e007223f94"
server_key_iv: "4b7279be01a0b9d28dec7c124e3bc35b"
server_key_version: "encryptionkey@0"
I20260928 02:38:13.364301 18963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "7bd630221fd745ac850a95e04d3e21b8"
format_stamp: "Formatted at 2026-09-28 02:38:13 on dist-test-slave-fz47"
server_key: "72827aeb51628d9b1d46a9e007223f94"
server_key_iv: "4b7279be01a0b9d28dec7c124e3bc35b"
server_key_version: "encryptionkey@0"
I20260928 02:38:13.365563 18963 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:38:13.366231 18977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:13.366375 18963 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:13.366456 18963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "7bd630221fd745ac850a95e04d3e21b8"
format_stamp: "Formatted at 2026-09-28 02:38:13 on dist-test-slave-fz47"
server_key: "72827aeb51628d9b1d46a9e007223f94"
server_key_iv: "4b7279be01a0b9d28dec7c124e3bc35b"
server_key_version: "encryptionkey@0"
I20260928 02:38:13.366520 18963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:13.389508 18963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:13.389820 18963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:13.389948 18963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:13.393611 18963 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39301
I20260928 02:38:13.393653 19029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39301 every 8 connection(s)
I20260928 02:38:13.393996 18963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:38:13.394894 19030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:13.397047 19030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bd630221fd745ac850a95e04d3e21b8: Bootstrap starting.
I20260928 02:38:13.397629 19030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bd630221fd745ac850a95e04d3e21b8: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:13.397869 19030 log.cc:824] T 00000000000000000000000000000000 P 7bd630221fd745ac850a95e04d3e21b8: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:13.398190 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 18963
I20260928 02:38:13.398288 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:38:13.398473 392 external_mini_cluster.cc:1428] Setting key 58a850c17b48a7b1376c83ca2d0815be
I20260928 02:38:13.398554 19030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bd630221fd745ac850a95e04d3e21b8: No bootstrap required, opened a new log
I20260928 02:38:13.400908 19030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bd630221fd745ac850a95e04d3e21b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd630221fd745ac850a95e04d3e21b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39301 } }
I20260928 02:38:13.401049 19030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bd630221fd745ac850a95e04d3e21b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:13.401082 19030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bd630221fd745ac850a95e04d3e21b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bd630221fd745ac850a95e04d3e21b8, State: Initialized, Role: FOLLOWER
I20260928 02:38:13.401217 19030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bd630221fd745ac850a95e04d3e21b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd630221fd745ac850a95e04d3e21b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39301 } }
I20260928 02:38:13.401316 19030 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bd630221fd745ac850a95e04d3e21b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:38:13.401351 19030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bd630221fd745ac850a95e04d3e21b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:38:13.401396 19030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bd630221fd745ac850a95e04d3e21b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:13.402036 19030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bd630221fd745ac850a95e04d3e21b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd630221fd745ac850a95e04d3e21b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39301 } }
I20260928 02:38:13.402135 19030 leader_election.cc:302] T 00000000000000000000000000000000 P 7bd630221fd745ac850a95e04d3e21b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bd630221fd745ac850a95e04d3e21b8; no voters:
I20260928 02:38:13.402295 19030 leader_election.cc:288] T 00000000000000000000000000000000 P 7bd630221fd745ac850a95e04d3e21b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:38:13.402333 19035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bd630221fd745ac850a95e04d3e21b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:13.402474 19035 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bd630221fd745ac850a95e04d3e21b8 [term 1 LEADER]: Becoming Leader. State: Replica: 7bd630221fd745ac850a95e04d3e21b8, State: Running, Role: LEADER
I20260928 02:38:13.402549 19030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bd630221fd745ac850a95e04d3e21b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:38:13.402612 19035 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bd630221fd745ac850a95e04d3e21b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd630221fd745ac850a95e04d3e21b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39301 } }
I20260928 02:38:13.402942 19035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bd630221fd745ac850a95e04d3e21b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bd630221fd745ac850a95e04d3e21b8. Latest consensus state: current_term: 1 leader_uuid: "7bd630221fd745ac850a95e04d3e21b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd630221fd745ac850a95e04d3e21b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39301 } } }
I20260928 02:38:13.403009 19035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bd630221fd745ac850a95e04d3e21b8 [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:13.403023 19036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bd630221fd745ac850a95e04d3e21b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bd630221fd745ac850a95e04d3e21b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd630221fd745ac850a95e04d3e21b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39301 } } }
I20260928 02:38:13.403138 19036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bd630221fd745ac850a95e04d3e21b8 [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:13.404254 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:13.398987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52538 (local address 127.0.98.62:39301)
0928 02:38:13.399164 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:13.399172 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:13.399197 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:38:13.399763 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:13.399770 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:13.399829 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:13.399938 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:13.400146 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.400992 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.402058 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.402627 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.402756 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:13.403613 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:13.403617 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:13.403617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:13.403824 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:13.403869 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:13.403884 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:13.403955 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:13.404035 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 02:38:13.404117 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
W20260928 02:38:13.404453 19051 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7bd630221fd745ac850a95e04d3e21b8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:38:13.404525 19051 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:38:13.404605 19039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:38:13.404774 19039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:38:13.406149 19039 catalog_manager.cc:1347] Generated new cluster ID: 9ab00d5986d846dfb49b880f08b8fb8b
I20260928 02:38:13.406204 19039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:38:13.412766 19039 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:38:13.413442 19039 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:38:13.423087 19039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bd630221fd745ac850a95e04d3e21b8: Generated new TSK 0
I20260928 02:38:13.423259 19039 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:38:13.431790 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39301
--builtin_ntp_servers=127.0.98.20:46447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:38:13.509208 19054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:13.509443 19054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:13.509472 19054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:13.510574 19054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:13.510618 19054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:13.510650 19054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:38:13.511946 19054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:13.512164 19054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:13.512418 19054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:13.517978 19059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:13.518144 19054 server_base.cc:1034] running on GCE node
W20260928 02:38:13.517964 19060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:13.517940 19062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:13.518456 19054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:13.518728 19054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:13.519909 19054 hybrid_clock.cc:648] HybridClock initialized: now 1790563093519889 us; error 32 us; skew 500 ppm
I20260928 02:38:13.521217 19054 webserver.cc:460] Webserver started at http://127.0.98.1:41871/ using document root <none> and password file <none>
I20260928 02:38:13.521482 19054 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:13.521543 19054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:13.521678 19054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:13.523058 19054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-0/data/instance:
uuid: "0c2a0cbe54154ce8a699517723a0663d"
format_stamp: "Formatted at 2026-09-28 02:38:13 on dist-test-slave-fz47"
server_key: "976a820dcce31151178b95720bf5f8e6"
server_key_iv: "1315e4c0425d7aecbc3e255fd2b243f2"
server_key_version: "encryptionkey@0"
I20260928 02:38:13.523458 19054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0c2a0cbe54154ce8a699517723a0663d"
format_stamp: "Formatted at 2026-09-28 02:38:13 on dist-test-slave-fz47"
server_key: "976a820dcce31151178b95720bf5f8e6"
server_key_iv: "1315e4c0425d7aecbc3e255fd2b243f2"
server_key_version: "encryptionkey@0"
I20260928 02:38:13.524937 19054 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:38:13.525748 19068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:13.525903 19054 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:13.525985 19054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "0c2a0cbe54154ce8a699517723a0663d"
format_stamp: "Formatted at 2026-09-28 02:38:13 on dist-test-slave-fz47"
server_key: "976a820dcce31151178b95720bf5f8e6"
server_key_iv: "1315e4c0425d7aecbc3e255fd2b243f2"
server_key_version: "encryptionkey@0"
I20260928 02:38:13.526047 19054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:13.541451 19054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:13.541703 19054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:13.541800 19054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:13.541968 19054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:38:13.542225 19054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:38:13.542253 19054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:13.542277 19054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:38:13.542294 19054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:13.547438 19054 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36321
I20260928 02:38:13.547515 19181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36321 every 8 connection(s)
I20260928 02:38:13.547784 19054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
I20260928 02:38:13.552428 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:13.548448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33017 (local address 127.0.98.62:39301)
0928 02:38:13.548489 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:13.548491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:13.548707 (+ 216us) server_negotiation.cc:407] Connection header received
0928 02:38:13.549187 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:13.549188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:13.549200 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:13.549251 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:13.549889 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.550369 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.551286 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.551471 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.551523 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:13.552212 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:13.552213 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:13.552213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:13.552217 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:13.552232 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:13.552235 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:13.552285 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:13.552350 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 02:38:13.552370 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:38:13.552613 19184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:13.548432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39301 (local address 127.0.98.1:33017)
0928 02:38:13.548598 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:13.548630 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:13.549105 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:13.549340 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:13.549343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:13.549353 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:13.549818 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:13.549822 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.550434 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.550435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:13.551211 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:13.551213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.551529 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.551530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:13.551627 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:13.552067 (+ 440us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:38:13.552067 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:13.552093 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:38:13.552115 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:13.552290 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:13.552292 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:13.552294 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:38:13.552339 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 02:38:13.552414 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260928 02:38:13.553151 19182 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39301
I20260928 02:38:13.553236 19182 heartbeater.cc:464] Registering TS with master...
I20260928 02:38:13.553426 19182 heartbeater.cc:511] Master 127.0.98.62:39301 requested a full tablet report, sending...
I20260928 02:38:13.553874 18994 ts_manager.cc:194] Registered new tserver with Master: 0c2a0cbe54154ce8a699517723a0663d (127.0.98.1:36321)
I20260928 02:38:13.554484 18994 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:33017
I20260928 02:38:13.555999 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 19054
I20260928 02:38:13.556097 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-0/wal/instance
I20260928 02:38:13.556273 392 external_mini_cluster.cc:1428] Setting key bd40a827e6c93b7b3da1bf5821dfd2cc
I20260928 02:38:13.557068 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39301
--builtin_ntp_servers=127.0.98.20:46447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:38:13.634418 19185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:13.634611 19185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:13.634630 19185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:13.635761 19185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:13.635800 19185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:13.635825 19185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:38:13.637156 19185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:13.637420 19185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:13.637832 19185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:13.642632 19193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:13.642632 19191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:13.642635 19190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:13.642647 19185 server_base.cc:1034] running on GCE node
I20260928 02:38:13.642985 19185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:13.643195 19185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:13.644349 19185 hybrid_clock.cc:648] HybridClock initialized: now 1790563093644334 us; error 23 us; skew 500 ppm
I20260928 02:38:13.648764 19185 webserver.cc:460] Webserver started at http://127.0.98.2:40641/ using document root <none> and password file <none>
I20260928 02:38:13.648984 19185 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:13.649032 19185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:13.649163 19185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:13.650238 19185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-1/data/instance:
uuid: "411698947a1649ea96a2d025794ad805"
format_stamp: "Formatted at 2026-09-28 02:38:13 on dist-test-slave-fz47"
server_key: "06d5f4d87c7c70b933ec58c272eff642"
server_key_iv: "bf7643acd79025f9a105202f6436ff34"
server_key_version: "encryptionkey@0"
I20260928 02:38:13.650548 19185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-1/wal/instance:
uuid: "411698947a1649ea96a2d025794ad805"
format_stamp: "Formatted at 2026-09-28 02:38:13 on dist-test-slave-fz47"
server_key: "06d5f4d87c7c70b933ec58c272eff642"
server_key_iv: "bf7643acd79025f9a105202f6436ff34"
server_key_version: "encryptionkey@0"
I20260928 02:38:13.651716 19185 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:38:13.652323 19199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:13.652467 19185 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:13.652544 19185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "411698947a1649ea96a2d025794ad805"
format_stamp: "Formatted at 2026-09-28 02:38:13 on dist-test-slave-fz47"
server_key: "06d5f4d87c7c70b933ec58c272eff642"
server_key_iv: "bf7643acd79025f9a105202f6436ff34"
server_key_version: "encryptionkey@0"
I20260928 02:38:13.652606 19185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:13.672873 19185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:13.673162 19185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:13.673313 19185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:13.673530 19185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:38:13.673839 19185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:38:13.673872 19185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:13.673905 19185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:38:13.673928 19185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:13.679199 19185 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34395
I20260928 02:38:13.679270 19312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34395 every 8 connection(s)
I20260928 02:38:13.679553 19185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
I20260928 02:38:13.681291 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 19185
I20260928 02:38:13.681388 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-1/wal/instance
I20260928 02:38:13.681597 392 external_mini_cluster.cc:1428] Setting key 2cffdef256565a9319c672e858c5dc68
I20260928 02:38:13.682533 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39301
--builtin_ntp_servers=127.0.98.20:46447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:38:13.685456 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:13.680237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54515 (local address 127.0.98.62:39301)
0928 02:38:13.680296 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:13.680297 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:13.680502 (+ 205us) server_negotiation.cc:407] Connection header received
0928 02:38:13.681021 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:13.681022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:13.681035 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:13.681099 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:13.681915 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.682599 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.683936 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.684225 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.684292 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:13.685193 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:13.685194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:13.685194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:13.685198 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:13.685211 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:13.685214 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:13.685288 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:13.685349 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 02:38:13.685370 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 02:38:13.685650 19315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:13.680206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39301 (local address 127.0.98.2:54515)
0928 02:38:13.680393 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:13.680426 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:13.680924 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:13.681175 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:13.681179 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:13.681188 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:13.681771 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:13.681777 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.682705 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.682706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:13.683852 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:13.683855 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.684303 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.684304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:13.684407 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:13.684888 (+ 481us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:38:13.684889 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:13.685067 (+ 178us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:38:13.685112 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:13.685278 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:13.685282 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:13.685285 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:38:13.685350 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 02:38:13.685449 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
I20260928 02:38:13.686088 19313 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39301
I20260928 02:38:13.686180 19313 heartbeater.cc:464] Registering TS with master...
I20260928 02:38:13.686362 19313 heartbeater.cc:511] Master 127.0.98.62:39301 requested a full tablet report, sending...
I20260928 02:38:13.686741 18994 ts_manager.cc:194] Registered new tserver with Master: 411698947a1649ea96a2d025794ad805 (127.0.98.2:34395)
I20260928 02:38:13.687091 18994 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:54515
W20260928 02:38:13.760459 19316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:13.760650 19316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:13.760670 19316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:13.761828 19316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:13.761868 19316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:13.761893 19316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:38:13.763219 19316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:13.763427 19316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:13.763692 19316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:13.768232 19322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:13.768240 19321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:13.768240 19324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:13.768484 19316 server_base.cc:1034] running on GCE node
I20260928 02:38:13.768743 19316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:13.768946 19316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:13.770095 19316 hybrid_clock.cc:648] HybridClock initialized: now 1790563093770085 us; error 27 us; skew 500 ppm
I20260928 02:38:13.771219 19316 webserver.cc:460] Webserver started at http://127.0.98.3:40277/ using document root <none> and password file <none>
I20260928 02:38:13.771422 19316 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:13.771466 19316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:13.771586 19316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:13.772603 19316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-2/data/instance:
uuid: "7fa7e50ca53d42f4b8d3042154bf5372"
format_stamp: "Formatted at 2026-09-28 02:38:13 on dist-test-slave-fz47"
server_key: "f498a59f5bce554a960b419587d35001"
server_key_iv: "3624164f5c35b6a646b96b557da15906"
server_key_version: "encryptionkey@0"
I20260928 02:38:13.772921 19316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7fa7e50ca53d42f4b8d3042154bf5372"
format_stamp: "Formatted at 2026-09-28 02:38:13 on dist-test-slave-fz47"
server_key: "f498a59f5bce554a960b419587d35001"
server_key_iv: "3624164f5c35b6a646b96b557da15906"
server_key_version: "encryptionkey@0"
I20260928 02:38:13.774111 19316 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:38:13.774718 19330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:13.774864 19316 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:38:13.774937 19316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "7fa7e50ca53d42f4b8d3042154bf5372"
format_stamp: "Formatted at 2026-09-28 02:38:13 on dist-test-slave-fz47"
server_key: "f498a59f5bce554a960b419587d35001"
server_key_iv: "3624164f5c35b6a646b96b557da15906"
server_key_version: "encryptionkey@0"
I20260928 02:38:13.775018 19316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:13.791229 19316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:13.791497 19316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:13.791618 19316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:13.791808 19316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:38:13.792111 19316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:38:13.792143 19316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:13.792169 19316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:38:13.792186 19316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:13.797411 19316 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46251
I20260928 02:38:13.797454 19443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46251 every 8 connection(s)
I20260928 02:38:13.797742 19316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
I20260928 02:38:13.802366 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:13.798424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37723 (local address 127.0.98.62:39301)
0928 02:38:13.798496 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:13.798499 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:13.798690 (+ 191us) server_negotiation.cc:407] Connection header received
0928 02:38:13.799140 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:13.799141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:13.799150 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:13.799197 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:13.799792 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.800264 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.801143 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.801363 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.801411 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:13.802111 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:13.802112 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:13.802112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:13.802116 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:13.802128 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:13.802132 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:13.802168 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:13.802260 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 02:38:13.802287 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:38:13.802547 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:13.798401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39301 (local address 127.0.98.3:37723)
0928 02:38:13.798583 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:13.798615 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:13.799068 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:13.799256 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:13.799260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:13.799267 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:13.799737 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:13.799740 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.800323 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.800323 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:13.801084 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:13.801086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.801424 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.801425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:13.801533 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:13.801979 (+ 446us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:38:13.801980 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:13.802016 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:38:13.802042 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:13.802176 (+ 134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:13.802180 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:13.802183 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:38:13.802248 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 02:38:13.802344 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
I20260928 02:38:13.802983 19444 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39301
I20260928 02:38:13.803066 19444 heartbeater.cc:464] Registering TS with master...
I20260928 02:38:13.803313 19444 heartbeater.cc:511] Master 127.0.98.62:39301 requested a full tablet report, sending...
I20260928 02:38:13.803699 18994 ts_manager.cc:194] Registered new tserver with Master: 7fa7e50ca53d42f4b8d3042154bf5372 (127.0.98.3:46251)
I20260928 02:38:13.804047 18994 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:37723
I20260928 02:38:13.807562 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 19316
I20260928 02:38:13.807638 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-2/wal/instance
I20260928 02:38:13.807801 392 external_mini_cluster.cc:1428] Setting key deb28fb571e47f60bc216bbfadf97a2b
I20260928 02:38:13.808614 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:38:13.812517 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:13.809865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52546 (local address 127.0.98.62:39301)
0928 02:38:13.809921 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:13.809923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:13.810017 (+ 94us) server_negotiation.cc:407] Connection header received
0928 02:38:13.810076 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:13.810077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:13.810086 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:13.810121 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:13.810267 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.810728 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.811437 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.811631 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.811681 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:13.812258 (+ 577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:13.812259 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:13.812259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:13.812264 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:13.812277 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:13.812280 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:13.812355 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:13.812433 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 02:38:13.812443 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 02:38:13.813948 18994 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:38:13.815372 18994 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:38:13.821183 19453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:13.817321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60716 (local address 127.0.98.3:46251)
0928 02:38:13.817464 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:13.817468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:13.817727 (+ 259us) server_negotiation.cc:407] Connection header received
0928 02:38:13.817807 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:13.817809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:13.817825 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:13.817896 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:13.818107 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.818816 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.819975 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.820676 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.820714 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:13.820812 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:13.821062 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 02:38:13.821101 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260928 02:38:13.821184 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:13.817241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36321 (local address 127.0.0.1:41124)
0928 02:38:13.817751 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:13.817759 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:13.817831 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:13.818073 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:13.818073 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:13.818075 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:13.818105 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:13.818106 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.819388 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.819389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:13.820222 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:13.820224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.820957 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.820958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:13.821015 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:13.821019 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:38:13.821068 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 02:38:13.821119 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":24,"spinlock_wait_cycles":876288,"thread_start_us":460,"threads_started":1}
I20260928 02:38:13.821199 19455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:13.817425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41124 (local address 127.0.98.1:36321)
0928 02:38:13.817610 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:13.817614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:13.817807 (+ 193us) server_negotiation.cc:407] Connection header received
0928 02:38:13.817913 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:13.817915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:13.817930 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:13.817997 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:13.818183 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.819296 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.820289 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.820864 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.820940 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:13.821029 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:13.821083 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 02:38:13.821120 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":99,"threads_started":1}
I20260928 02:38:13.821569 19452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:13.817235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46251 (local address 127.0.0.1:60716)
0928 02:38:13.817620 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:13.817656 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:13.817731 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:13.817963 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:13.817966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:13.817968 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:13.818028 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:13.818030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.818896 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.818897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:13.819880 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:13.819883 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.820870 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.820871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:13.820948 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:13.820961 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 02:38:13.821457 (+ 496us) client_negotiation.cc:240] Negotiation successful
0928 02:38:13.821511 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":71,"threads_started":1}
I20260928 02:38:13.821727 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:13.817396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41260 (local address 127.0.98.2:34395)
0928 02:38:13.817581 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:13.817585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:13.817811 (+ 226us) server_negotiation.cc:407] Connection header received
0928 02:38:13.817852 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:13.817854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:13.817868 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:13.818060 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:13.818212 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.819296 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.820574 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.821238 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.821318 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:13.821399 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:13.821606 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 02:38:13.821652 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":114,"threads_started":1}
I20260928 02:38:13.821667 19456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:13.817240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34395 (local address 127.0.0.1:41260)
0928 02:38:13.817678 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:13.817687 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:13.817743 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:13.817961 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:13.817970 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:13.817972 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:13.818135 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:13.818136 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.819383 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.819384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:13.820495 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:13.820497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.821430 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.821431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:13.821504 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:13.821508 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:38:13.821563 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 02:38:13.821603 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":22,"thread_start_us":399,"threads_started":1}
I20260928 02:38:13.822067 19116 tablet_service.cc:1467] Processing CreateTablet for tablet 9f10d1d783af47ca96388bab8fe2a80f (DEFAULT_TABLE table=test-table [id=0d6d45b55e404420af865138d6e3c49d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:13.822312 19378 tablet_service.cc:1467] Processing CreateTablet for tablet 9f10d1d783af47ca96388bab8fe2a80f (DEFAULT_TABLE table=test-table [id=0d6d45b55e404420af865138d6e3c49d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:13.822450 19116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f10d1d783af47ca96388bab8fe2a80f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:13.822501 19247 tablet_service.cc:1467] Processing CreateTablet for tablet 9f10d1d783af47ca96388bab8fe2a80f (DEFAULT_TABLE table=test-table [id=0d6d45b55e404420af865138d6e3c49d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:13.822614 19378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f10d1d783af47ca96388bab8fe2a80f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:13.822784 19247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f10d1d783af47ca96388bab8fe2a80f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:13.824846 19458 tablet_bootstrap.cc:492] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372: Bootstrap starting.
I20260928 02:38:13.825428 19458 tablet_bootstrap.cc:654] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:13.825601 19459 tablet_bootstrap.cc:492] T 9f10d1d783af47ca96388bab8fe2a80f P 0c2a0cbe54154ce8a699517723a0663d: Bootstrap starting.
I20260928 02:38:13.825719 19458 log.cc:824] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:13.825973 19460 tablet_bootstrap.cc:492] T 9f10d1d783af47ca96388bab8fe2a80f P 411698947a1649ea96a2d025794ad805: Bootstrap starting.
I20260928 02:38:13.826161 19459 tablet_bootstrap.cc:654] T 9f10d1d783af47ca96388bab8fe2a80f P 0c2a0cbe54154ce8a699517723a0663d: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:13.826412 19459 log.cc:824] T 9f10d1d783af47ca96388bab8fe2a80f P 0c2a0cbe54154ce8a699517723a0663d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:13.826498 19460 tablet_bootstrap.cc:654] T 9f10d1d783af47ca96388bab8fe2a80f P 411698947a1649ea96a2d025794ad805: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:13.826794 19460 log.cc:824] T 9f10d1d783af47ca96388bab8fe2a80f P 411698947a1649ea96a2d025794ad805: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:13.827054 19459 tablet_bootstrap.cc:492] T 9f10d1d783af47ca96388bab8fe2a80f P 0c2a0cbe54154ce8a699517723a0663d: No bootstrap required, opened a new log
I20260928 02:38:13.827111 19459 ts_tablet_manager.cc:1397] T 9f10d1d783af47ca96388bab8fe2a80f P 0c2a0cbe54154ce8a699517723a0663d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:38:13.827387 19460 tablet_bootstrap.cc:492] T 9f10d1d783af47ca96388bab8fe2a80f P 411698947a1649ea96a2d025794ad805: No bootstrap required, opened a new log
I20260928 02:38:13.827435 19460 ts_tablet_manager.cc:1397] T 9f10d1d783af47ca96388bab8fe2a80f P 411698947a1649ea96a2d025794ad805: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:38:13.828972 19459 raft_consensus.cc:348] T 9f10d1d783af47ca96388bab8fe2a80f P 0c2a0cbe54154ce8a699517723a0663d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa7e50ca53d42f4b8d3042154bf5372" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46251 } } peers { permanent_uuid: "0c2a0cbe54154ce8a699517723a0663d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "411698947a1649ea96a2d025794ad805" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } }
I20260928 02:38:13.829108 19459 raft_consensus.cc:374] T 9f10d1d783af47ca96388bab8fe2a80f P 0c2a0cbe54154ce8a699517723a0663d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:13.829138 19459 raft_consensus.cc:729] T 9f10d1d783af47ca96388bab8fe2a80f P 0c2a0cbe54154ce8a699517723a0663d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c2a0cbe54154ce8a699517723a0663d, State: Initialized, Role: FOLLOWER
I20260928 02:38:13.829248 19459 consensus_queue.cc:260] T 9f10d1d783af47ca96388bab8fe2a80f P 0c2a0cbe54154ce8a699517723a0663d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa7e50ca53d42f4b8d3042154bf5372" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46251 } } peers { permanent_uuid: "0c2a0cbe54154ce8a699517723a0663d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "411698947a1649ea96a2d025794ad805" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } }
I20260928 02:38:13.829356 19460 raft_consensus.cc:348] T 9f10d1d783af47ca96388bab8fe2a80f P 411698947a1649ea96a2d025794ad805 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa7e50ca53d42f4b8d3042154bf5372" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46251 } } peers { permanent_uuid: "0c2a0cbe54154ce8a699517723a0663d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "411698947a1649ea96a2d025794ad805" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } }
I20260928 02:38:13.829515 19460 raft_consensus.cc:374] T 9f10d1d783af47ca96388bab8fe2a80f P 411698947a1649ea96a2d025794ad805 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:13.829519 19459 ts_tablet_manager.cc:1428] T 9f10d1d783af47ca96388bab8fe2a80f P 0c2a0cbe54154ce8a699517723a0663d: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 02:38:13.829594 19460 raft_consensus.cc:729] T 9f10d1d783af47ca96388bab8fe2a80f P 411698947a1649ea96a2d025794ad805 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 411698947a1649ea96a2d025794ad805, State: Initialized, Role: FOLLOWER
I20260928 02:38:13.829607 19458 tablet_bootstrap.cc:492] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372: No bootstrap required, opened a new log
I20260928 02:38:13.829643 19182 heartbeater.cc:503] Master 127.0.98.62:39301 was elected leader, sending a full tablet report...
I20260928 02:38:13.829662 19458 ts_tablet_manager.cc:1397] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 02:38:13.829738 19460 consensus_queue.cc:260] T 9f10d1d783af47ca96388bab8fe2a80f P 411698947a1649ea96a2d025794ad805 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa7e50ca53d42f4b8d3042154bf5372" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46251 } } peers { permanent_uuid: "0c2a0cbe54154ce8a699517723a0663d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "411698947a1649ea96a2d025794ad805" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } }
I20260928 02:38:13.829978 19460 ts_tablet_manager.cc:1428] T 9f10d1d783af47ca96388bab8fe2a80f P 411698947a1649ea96a2d025794ad805: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:38:13.830065 19313 heartbeater.cc:503] Master 127.0.98.62:39301 was elected leader, sending a full tablet report...
I20260928 02:38:13.831754 19458 raft_consensus.cc:348] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa7e50ca53d42f4b8d3042154bf5372" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46251 } } peers { permanent_uuid: "0c2a0cbe54154ce8a699517723a0663d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "411698947a1649ea96a2d025794ad805" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } }
I20260928 02:38:13.831868 19458 raft_consensus.cc:374] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:13.831897 19458 raft_consensus.cc:729] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa7e50ca53d42f4b8d3042154bf5372, State: Initialized, Role: FOLLOWER
I20260928 02:38:13.832072 19458 consensus_queue.cc:260] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa7e50ca53d42f4b8d3042154bf5372" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46251 } } peers { permanent_uuid: "0c2a0cbe54154ce8a699517723a0663d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "411698947a1649ea96a2d025794ad805" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } }
I20260928 02:38:13.832300 19458 ts_tablet_manager.cc:1428] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:38:13.832337 19444 heartbeater.cc:503] Master 127.0.98.62:39301 was elected leader, sending a full tablet report...
I20260928 02:38:13.877419 19466 raft_consensus.cc:482] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:38:13.877571 19466 raft_consensus.cc:504] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa7e50ca53d42f4b8d3042154bf5372" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46251 } } peers { permanent_uuid: "0c2a0cbe54154ce8a699517723a0663d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "411698947a1649ea96a2d025794ad805" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } }
I20260928 02:38:13.877925 19466 leader_election.cc:288] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c2a0cbe54154ce8a699517723a0663d (127.0.98.1:36321), 411698947a1649ea96a2d025794ad805 (127.0.98.2:34395)
I20260928 02:38:13.880625 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:13.878186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48201 (local address 127.0.98.2:34395)
0928 02:38:13.878259 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:13.878262 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:13.878346 (+ 84us) server_negotiation.cc:407] Connection header received
0928 02:38:13.878400 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:13.878401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:13.878409 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:13.878464 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:13.878602 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.879053 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.879975 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.880318 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.880376 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:13.880387 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:13.880527 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 02:38:13.880562 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:38:13.880635 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:13.878077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36321 (local address 127.0.98.3:35339)
0928 02:38:13.878148 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:13.878160 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:13.878208 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:13.878377 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:13.878377 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:13.878379 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:13.878452 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:13.878455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.879104 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.879105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:13.879876 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:13.879878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.880394 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.880394 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:13.880471 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:13.880473 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:38:13.880519 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 02:38:13.880558 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 02:38:13.880635 19455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:13.878215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35339 (local address 127.0.98.1:36321)
0928 02:38:13.878262 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:13.878264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:13.878273 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:38:13.878302 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:13.878302 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:13.878313 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:13.878366 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:13.878533 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.879035 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.879966 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.880324 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.880382 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:13.880390 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:13.880538 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 02:38:13.880574 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 02:38:13.880646 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:13.878104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34395 (local address 127.0.98.3:48201)
0928 02:38:13.878282 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:13.878291 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:13.878347 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:13.878465 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:13.878466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:13.878467 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:13.878533 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:13.878535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.879123 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.879124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:13.879881 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:13.879883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.880387 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.880387 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:13.880460 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:13.880464 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:38:13.880509 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 02:38:13.880546 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":18,"thread_start_us":121,"threads_started":1}
I20260928 02:38:13.880983 19136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f10d1d783af47ca96388bab8fe2a80f" candidate_uuid: "7fa7e50ca53d42f4b8d3042154bf5372" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c2a0cbe54154ce8a699517723a0663d" is_pre_election: true
I20260928 02:38:13.881143 19136 raft_consensus.cc:2454] T 9f10d1d783af47ca96388bab8fe2a80f P 0c2a0cbe54154ce8a699517723a0663d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa7e50ca53d42f4b8d3042154bf5372 in term 0.
I20260928 02:38:13.881361 19332 leader_election.cc:302] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c2a0cbe54154ce8a699517723a0663d, 7fa7e50ca53d42f4b8d3042154bf5372; no voters:
I20260928 02:38:13.881467 19267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f10d1d783af47ca96388bab8fe2a80f" candidate_uuid: "7fa7e50ca53d42f4b8d3042154bf5372" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "411698947a1649ea96a2d025794ad805" is_pre_election: true
I20260928 02:38:13.881495 19466 raft_consensus.cc:2781] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:38:13.881556 19466 raft_consensus.cc:482] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:38:13.881587 19466 raft_consensus.cc:3037] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:13.881590 19267 raft_consensus.cc:2454] T 9f10d1d783af47ca96388bab8fe2a80f P 411698947a1649ea96a2d025794ad805 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa7e50ca53d42f4b8d3042154bf5372 in term 0.
I20260928 02:38:13.882283 19466 raft_consensus.cc:504] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa7e50ca53d42f4b8d3042154bf5372" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46251 } } peers { permanent_uuid: "0c2a0cbe54154ce8a699517723a0663d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "411698947a1649ea96a2d025794ad805" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } }
I20260928 02:38:13.882406 19466 leader_election.cc:288] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372 [CANDIDATE]: Term 1 election: Requested vote from peers 0c2a0cbe54154ce8a699517723a0663d (127.0.98.1:36321), 411698947a1649ea96a2d025794ad805 (127.0.98.2:34395)
I20260928 02:38:13.882548 19267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f10d1d783af47ca96388bab8fe2a80f" candidate_uuid: "7fa7e50ca53d42f4b8d3042154bf5372" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "411698947a1649ea96a2d025794ad805"
I20260928 02:38:13.882620 19267 raft_consensus.cc:3037] T 9f10d1d783af47ca96388bab8fe2a80f P 411698947a1649ea96a2d025794ad805 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:13.883303 19267 raft_consensus.cc:2454] T 9f10d1d783af47ca96388bab8fe2a80f P 411698947a1649ea96a2d025794ad805 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa7e50ca53d42f4b8d3042154bf5372 in term 1.
I20260928 02:38:13.883402 19136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f10d1d783af47ca96388bab8fe2a80f" candidate_uuid: "7fa7e50ca53d42f4b8d3042154bf5372" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c2a0cbe54154ce8a699517723a0663d"
I20260928 02:38:13.883448 19334 leader_election.cc:302] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 411698947a1649ea96a2d025794ad805, 7fa7e50ca53d42f4b8d3042154bf5372; no voters:
I20260928 02:38:13.883481 19136 raft_consensus.cc:3037] T 9f10d1d783af47ca96388bab8fe2a80f P 0c2a0cbe54154ce8a699517723a0663d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:13.883562 19466 raft_consensus.cc:2781] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:13.883718 19466 raft_consensus.cc:686] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372 [term 1 LEADER]: Becoming Leader. State: Replica: 7fa7e50ca53d42f4b8d3042154bf5372, State: Running, Role: LEADER
I20260928 02:38:13.883798 19466 consensus_queue.cc:237] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa7e50ca53d42f4b8d3042154bf5372" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46251 } } peers { permanent_uuid: "0c2a0cbe54154ce8a699517723a0663d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "411698947a1649ea96a2d025794ad805" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } }
I20260928 02:38:13.884323 19136 raft_consensus.cc:2454] T 9f10d1d783af47ca96388bab8fe2a80f P 0c2a0cbe54154ce8a699517723a0663d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa7e50ca53d42f4b8d3042154bf5372 in term 1.
I20260928 02:38:13.884617 18994 catalog_manager.cc:5495] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fa7e50ca53d42f4b8d3042154bf5372 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "7fa7e50ca53d42f4b8d3042154bf5372" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fa7e50ca53d42f4b8d3042154bf5372" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c2a0cbe54154ce8a699517723a0663d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "411698947a1649ea96a2d025794ad805" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 } health_report { overall_health: UNKNOWN } } }
I20260928 02:38:13.896246 19453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:13.894261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60726 (local address 127.0.98.3:46251)
0928 02:38:13.894313 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:13.894314 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:13.894322 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:38:13.894394 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:13.894395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:13.894410 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:13.894457 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:38:13.894572 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.895007 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.895660 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:13.895831 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:13.895873 (+ 42us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:13.895976 (+ 103us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:38:13.896053 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:38:13.896119 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:13.896152 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 02:38:13.896159 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 02:38:13.897949 19136 raft_consensus.cc:1261] T 9f10d1d783af47ca96388bab8fe2a80f P 0c2a0cbe54154ce8a699517723a0663d [term 1 FOLLOWER]: Refusing update from remote peer 7fa7e50ca53d42f4b8d3042154bf5372: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:38:13.897949 19267 raft_consensus.cc:1261] T 9f10d1d783af47ca96388bab8fe2a80f P 411698947a1649ea96a2d025794ad805 [term 1 FOLLOWER]: Refusing update from remote peer 7fa7e50ca53d42f4b8d3042154bf5372: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:38:13.898291 19468 consensus_queue.cc:1035] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "411698947a1649ea96a2d025794ad805" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:38:13.898407 19466 consensus_queue.cc:1035] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c2a0cbe54154ce8a699517723a0663d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:38:13.901685 19472 mvcc.cc:205] Tried to move back new op lower bound from 7334146432603602944 to 7334146432548818944. Current Snapshot: MvccSnapshot[applied={T|T < 7334146432603602944 or (T in {7334146432603602944})}]
I20260928 02:38:13.902804 18994 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:38:13.902891 18994 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:38:13.903576 18994 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7bd630221fd745ac850a95e04d3e21b8: Sending DeleteTablet for 3 replicas of tablet 9f10d1d783af47ca96388bab8fe2a80f
I20260928 02:38:13.903873 19378 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f10d1d783af47ca96388bab8fe2a80f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:13 UTC) from {username='slave'} at 127.0.0.1:60716
I20260928 02:38:13.903882 19247 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f10d1d783af47ca96388bab8fe2a80f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:13 UTC) from {username='slave'} at 127.0.0.1:41260
I20260928 02:38:13.904304 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 19054
I20260928 02:38:13.904346 19116 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f10d1d783af47ca96388bab8fe2a80f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:13 UTC) from {username='slave'} at 127.0.0.1:41124
I20260928 02:38:13.906518 19480 tablet_replica.cc:330] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372: stopping tablet replica
I20260928 02:38:13.906622 19480 raft_consensus.cc:2229] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:38:13.906713 19480 raft_consensus.cc:2258] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:38:13.906839 19481 tablet_replica.cc:330] T 9f10d1d783af47ca96388bab8fe2a80f P 411698947a1649ea96a2d025794ad805: stopping tablet replica
I20260928 02:38:13.906975 19481 raft_consensus.cc:2229] T 9f10d1d783af47ca96388bab8fe2a80f P 411698947a1649ea96a2d025794ad805 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:38:13.907130 19481 raft_consensus.cc:2258] T 9f10d1d783af47ca96388bab8fe2a80f P 411698947a1649ea96a2d025794ad805 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:38:13.907163 19480 ts_tablet_manager.cc:1905] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:38:13.907588 19481 ts_tablet_manager.cc:1905] T 9f10d1d783af47ca96388bab8fe2a80f P 411698947a1649ea96a2d025794ad805: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:38:13.908777 19480 ts_tablet_manager.cc:1918] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:38:13.908885 19480 log.cc:1196] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-2/wal/wals/9f10d1d783af47ca96388bab8fe2a80f
I20260928 02:38:13.909157 19481 ts_tablet_manager.cc:1918] T 9f10d1d783af47ca96388bab8fe2a80f P 411698947a1649ea96a2d025794ad805: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:38:13.909173 19480 ts_tablet_manager.cc:1939] T 9f10d1d783af47ca96388bab8fe2a80f P 7fa7e50ca53d42f4b8d3042154bf5372: Deleting consensus metadata
I20260928 02:38:13.909312 19481 log.cc:1196] T 9f10d1d783af47ca96388bab8fe2a80f P 411698947a1649ea96a2d025794ad805: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790562862804661-392-0/minicluster-data/ts-1/wal/wals/9f10d1d783af47ca96388bab8fe2a80f
I20260928 02:38:13.909554 18978 catalog_manager.cc:4841] TS 7fa7e50ca53d42f4b8d3042154bf5372 (127.0.98.3:46251): tablet 9f10d1d783af47ca96388bab8fe2a80f (table test-table [id=0d6d45b55e404420af865138d6e3c49d]) successfully deleted
I20260928 02:38:13.909583 19481 ts_tablet_manager.cc:1939] T 9f10d1d783af47ca96388bab8fe2a80f P 411698947a1649ea96a2d025794ad805: Deleting consensus metadata
I20260928 02:38:13.909999 18981 catalog_manager.cc:4841] TS 411698947a1649ea96a2d025794ad805 (127.0.98.2:34395): tablet 9f10d1d783af47ca96388bab8fe2a80f (table test-table [id=0d6d45b55e404420af865138d6e3c49d]) successfully deleted
W20260928 02:38:13.910290 18979 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36321 (error 108)
I20260928 02:38:13.910694 19452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:13.910559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36321 (local address 127.0.0.1:41136)
0928 02:38:13.910614 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:13.910662 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 02:38:13.910770 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 19185
W20260928 02:38:13.910807 18979 catalog_manager.cc:4584] TS 0c2a0cbe54154ce8a699517723a0663d (127.0.98.1:36321): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9f10d1d783af47ca96388bab8fe2a80f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36321: connect: Connection refused (error 111)
I20260928 02:38:13.916993 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 19316
I20260928 02:38:13.921552 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 18963
2026-09-28T02:38:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (671 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:38:13 dist-test-slave-fz47 krb5kdc[19485](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:38:13 dist-test-slave-fz47 krb5kdc[19485](info): set up 2 sockets
Sep 28 02:38:13 dist-test-slave-fz47 krb5kdc[19485](info): commencing operation
krb5kdc: starting...
W20260928 02:38:14.860694 392 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:38:14 dist-test-slave-fz47 krb5kdc[19485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:38:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:38:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 02:38:14.968250 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35781
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:15.046090 19498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:15.046307 19498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:15.046339 19498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:15.047468 19498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:15.047508 19498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:15.047526 19498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:15.047542 19498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:15.047556 19498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:15.048995 19498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:15.049222 19498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:15.049453 19498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:15.054009 19503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:15.053999 19504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:15.054030 19506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:15.054195 19498 server_base.cc:1034] running on GCE node
I20260928 02:38:15.054446 19498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:15.054725 19498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:15.055861 19498 hybrid_clock.cc:648] HybridClock initialized: now 1790563095055847 us; error 28 us; skew 500 ppm
Sep 28 02:38:15 dist-test-slave-fz47 krb5kdc[19485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:15.058187 19498 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:38:15.058574 19498 webserver.cc:460] Webserver started at http://127.0.98.62:33425/ using document root <none> and password file <none>
I20260928 02:38:15.058784 19498 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:15.058830 19498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:15.058950 19498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:15.059952 19498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "92fc23b21ce44360aee72d658f7450e6"
format_stamp: "Formatted at 2026-09-28 02:38:15 on dist-test-slave-fz47"
server_key: "b2a38a5899ad949ca61c3c2e4511eb0a"
server_key_iv: "96ff380ce200e429887c6d9eb15167f8"
server_key_version: "encryptionkey@0"
I20260928 02:38:15.060283 19498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "92fc23b21ce44360aee72d658f7450e6"
format_stamp: "Formatted at 2026-09-28 02:38:15 on dist-test-slave-fz47"
server_key: "b2a38a5899ad949ca61c3c2e4511eb0a"
server_key_iv: "96ff380ce200e429887c6d9eb15167f8"
server_key_version: "encryptionkey@0"
I20260928 02:38:15.061501 19498 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:38:15.062170 19513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:15.062335 19498 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:15.062415 19498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "92fc23b21ce44360aee72d658f7450e6"
format_stamp: "Formatted at 2026-09-28 02:38:15 on dist-test-slave-fz47"
server_key: "b2a38a5899ad949ca61c3c2e4511eb0a"
server_key_iv: "96ff380ce200e429887c6d9eb15167f8"
server_key_version: "encryptionkey@0"
I20260928 02:38:15.062471 19498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:15.103797 19498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:15.105713 19498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:15.105829 19498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:15.109591 19498 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35781
I20260928 02:38:15.109635 19565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35781 every 8 connection(s)
I20260928 02:38:15.109956 19498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:38:15.110875 19566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:15.112557 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 19498
I20260928 02:38:15.112660 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:38:15.112840 392 external_mini_cluster.cc:1428] Setting key 9889a072b387beb68c3616046f3bc120
I20260928 02:38:15.113344 19566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92fc23b21ce44360aee72d658f7450e6: Bootstrap starting.
I20260928 02:38:15.113934 19566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92fc23b21ce44360aee72d658f7450e6: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:15.114205 19566 log.cc:824] T 00000000000000000000000000000000 P 92fc23b21ce44360aee72d658f7450e6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:15.114919 19566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92fc23b21ce44360aee72d658f7450e6: No bootstrap required, opened a new log
I20260928 02:38:15.116926 19566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92fc23b21ce44360aee72d658f7450e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92fc23b21ce44360aee72d658f7450e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35781 } }
I20260928 02:38:15.117062 19566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92fc23b21ce44360aee72d658f7450e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:15.117084 19566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92fc23b21ce44360aee72d658f7450e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92fc23b21ce44360aee72d658f7450e6, State: Initialized, Role: FOLLOWER
I20260928 02:38:15.117193 19566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92fc23b21ce44360aee72d658f7450e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92fc23b21ce44360aee72d658f7450e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35781 } }
I20260928 02:38:15.117278 19566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92fc23b21ce44360aee72d658f7450e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:38:15.117342 19566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92fc23b21ce44360aee72d658f7450e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:38:15.117393 19566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92fc23b21ce44360aee72d658f7450e6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:15.117977 19566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92fc23b21ce44360aee72d658f7450e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92fc23b21ce44360aee72d658f7450e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35781 } }
I20260928 02:38:15.118073 19566 leader_election.cc:302] T 00000000000000000000000000000000 P 92fc23b21ce44360aee72d658f7450e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92fc23b21ce44360aee72d658f7450e6; no voters:
I20260928 02:38:15.118218 19566 leader_election.cc:288] T 00000000000000000000000000000000 P 92fc23b21ce44360aee72d658f7450e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:38:15.118256 19571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92fc23b21ce44360aee72d658f7450e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:15.118403 19571 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92fc23b21ce44360aee72d658f7450e6 [term 1 LEADER]: Becoming Leader. State: Replica: 92fc23b21ce44360aee72d658f7450e6, State: Running, Role: LEADER
I20260928 02:38:15.118496 19566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92fc23b21ce44360aee72d658f7450e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:38:15.118536 19571 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92fc23b21ce44360aee72d658f7450e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92fc23b21ce44360aee72d658f7450e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35781 } }
I20260928 02:38:15.118943 19571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92fc23b21ce44360aee72d658f7450e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92fc23b21ce44360aee72d658f7450e6. Latest consensus state: current_term: 1 leader_uuid: "92fc23b21ce44360aee72d658f7450e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92fc23b21ce44360aee72d658f7450e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35781 } } }
I20260928 02:38:15.119078 19571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92fc23b21ce44360aee72d658f7450e6 [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:15.119256 19572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92fc23b21ce44360aee72d658f7450e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92fc23b21ce44360aee72d658f7450e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92fc23b21ce44360aee72d658f7450e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35781 } } }
I20260928 02:38:15.119362 19572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92fc23b21ce44360aee72d658f7450e6 [sys.catalog]: This master's current role is: LEADER
Sep 28 02:38:15 dist-test-slave-fz47 krb5kdc[19485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
W20260928 02:38:15.120586 19585 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 92fc23b21ce44360aee72d658f7450e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:38:15.120932 19585 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:38:15.121014 19587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:38:15.121161 19587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:38:15.122568 19587 catalog_manager.cc:1347] Generated new cluster ID: 68067e61b21c4e1096d14e12ab2e2aba
I20260928 02:38:15.122618 19587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:38:15.123301 19569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:15.113365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50098 (local address 127.0.98.62:35781)
0928 02:38:15.113533 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:15.113540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:15.113566 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:38:15.113999 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:15.114008 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:15.114070 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:15.114189 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:15.114852 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.115768 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.116830 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.117329 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.117430 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:15.120002 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:15.120006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:15.120007 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:15.120019 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:15.121742 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:15.122156 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:15.122159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:15.122160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:15.122287 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:15.122476 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:15.122477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:15.122477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:15.122766 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:15.122895 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:15.123014 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 02:38:15.123144 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":62,"threads_started":1}
I20260928 02:38:15.132275 19587 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:38:15.132958 19587 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:38:15.137466 19587 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92fc23b21ce44360aee72d658f7450e6: Generated new TSK 0
I20260928 02:38:15.137631 19587 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 02:38:15.161413 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35781
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:15.243470 19592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:15.243701 19592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:15.243727 19592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:15.245209 19592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:15.245306 19592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:15.245352 19592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:38:15.247221 19592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:15.247478 19592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:15.247717 19592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:15.252768 19597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:15.252768 19598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:15.252763 19600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:15.253521 19592 server_base.cc:1034] running on GCE node
I20260928 02:38:15.253742 19592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:15.253983 19592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:15.255122 19592 hybrid_clock.cc:648] HybridClock initialized: now 1790563095255111 us; error 31 us; skew 500 ppm
Sep 28 02:38:15 dist-test-slave-fz47 krb5kdc[19485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:15.257632 19592 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 02:38:15.258085 19592 webserver.cc:460] Webserver started at http://127.0.98.1:45747/ using document root <none> and password file <none>
I20260928 02:38:15.258302 19592 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:15.258347 19592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:15.258451 19592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:15.259467 19592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-0/data/instance:
uuid: "df84da30e48449828e76c728190ea9f8"
format_stamp: "Formatted at 2026-09-28 02:38:15 on dist-test-slave-fz47"
server_key: "14f1db05ffc4e91a1bd6d5c140cef01d"
server_key_iv: "55b2f6bada3f96010f28c0c12e2a9f77"
server_key_version: "encryptionkey@0"
I20260928 02:38:15.259783 19592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-0/wal/instance:
uuid: "df84da30e48449828e76c728190ea9f8"
format_stamp: "Formatted at 2026-09-28 02:38:15 on dist-test-slave-fz47"
server_key: "14f1db05ffc4e91a1bd6d5c140cef01d"
server_key_iv: "55b2f6bada3f96010f28c0c12e2a9f77"
server_key_version: "encryptionkey@0"
I20260928 02:38:15.261013 19592 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:38:15.261754 19607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:15.261922 19592 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:15.262009 19592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "df84da30e48449828e76c728190ea9f8"
format_stamp: "Formatted at 2026-09-28 02:38:15 on dist-test-slave-fz47"
server_key: "14f1db05ffc4e91a1bd6d5c140cef01d"
server_key_iv: "55b2f6bada3f96010f28c0c12e2a9f77"
server_key_version: "encryptionkey@0"
I20260928 02:38:15.262071 19592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:15.274403 19592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:15.276597 19592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:15.276746 19592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:15.276978 19592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:38:15.277313 19592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:38:15.277354 19592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:15.277385 19592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:38:15.277408 19592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:15.283013 19592 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36921
I20260928 02:38:15.283075 19720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36921 every 8 connection(s)
I20260928 02:38:15.283363 19592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
I20260928 02:38:15.287552 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 19592
I20260928 02:38:15.287663 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-0/wal/instance
I20260928 02:38:15.287881 392 external_mini_cluster.cc:1428] Setting key 3edbf12fd5eec33031fcffeb6ae4da37
Sep 28 02:38:15 dist-test-slave-fz47 krb5kdc[19485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:38:15.292874 19569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:15.284034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49077 (local address 127.0.98.62:35781)
0928 02:38:15.284110 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:15.284112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:15.284323 (+ 211us) server_negotiation.cc:407] Connection header received
0928 02:38:15.285229 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:15.285230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:15.285239 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:15.285322 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:15.285918 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.286421 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.287364 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.287572 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.287622 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:15.290182 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:15.290183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:15.290184 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:15.290192 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:15.291511 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:15.291962 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:15.291963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:15.291963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:15.292018 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:15.292261 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:15.292262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:15.292262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:15.292376 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:15.292436 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:15.292697 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 02:38:15.292786 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:38:15.293138 19723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:15.284005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35781 (local address 127.0.98.1:49077)
0928 02:38:15.284152 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:15.284234 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:15.285148 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:15.285397 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:15.285401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:15.285657 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:15.285851 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:15.285855 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.286490 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.286491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:15.287304 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:15.287305 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.287647 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.287648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:15.287884 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:15.288429 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:15.288439 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:15.290085 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:15.291588 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:15.291595 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:15.291597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:15.291884 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:15.292090 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:15.292091 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:15.292091 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:15.292202 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:15.292432 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:15.292439 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:15.292579 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 02:38:15.292671 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 02:38:15.292828 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":76,"threads_started":1}
I20260928 02:38:15.293685 19721 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35781
I20260928 02:38:15.293776 19721 heartbeater.cc:464] Registering TS with master...
I20260928 02:38:15.293947 19721 heartbeater.cc:511] Master 127.0.98.62:35781 requested a full tablet report, sending...
I20260928 02:38:15.294497 19530 ts_manager.cc:194] Registered new tserver with Master: df84da30e48449828e76c728190ea9f8 (127.0.98.1:36921)
I20260928 02:38:15.295421 19530 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49077
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 02:38:15.312166 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35781
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:15.393374 19726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:15.393572 19726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:15.393595 19726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:15.394699 19726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:15.394729 19726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:15.394754 19726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:38:15.396191 19726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:15.396382 19726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:15.396556 19726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:15.401216 19732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:15.401221 19731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:15.401290 19726 server_base.cc:1034] running on GCE node
W20260928 02:38:15.401221 19734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:15.401631 19726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:15.401827 19726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:15.402999 19726 hybrid_clock.cc:648] HybridClock initialized: now 1790563095402981 us; error 30 us; skew 500 ppm
Sep 28 02:38:15 dist-test-slave-fz47 krb5kdc[19485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:15.405349 19726 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 02:38:15.405733 19726 webserver.cc:460] Webserver started at http://127.0.98.2:46477/ using document root <none> and password file <none>
I20260928 02:38:15.405951 19726 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:15.405998 19726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:15.406091 19726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:15.407135 19726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-1/data/instance:
uuid: "7f176d6ab32649d39b829bc7679e6b28"
format_stamp: "Formatted at 2026-09-28 02:38:15 on dist-test-slave-fz47"
server_key: "acf1318994c8e48e51f95008fbffd8d5"
server_key_iv: "56193148049bd48bef2d00c68f606cc6"
server_key_version: "encryptionkey@0"
I20260928 02:38:15.407449 19726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7f176d6ab32649d39b829bc7679e6b28"
format_stamp: "Formatted at 2026-09-28 02:38:15 on dist-test-slave-fz47"
server_key: "acf1318994c8e48e51f95008fbffd8d5"
server_key_iv: "56193148049bd48bef2d00c68f606cc6"
server_key_version: "encryptionkey@0"
I20260928 02:38:15.408614 19726 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:38:15.409245 19741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:15.409389 19726 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:15.409483 19726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "7f176d6ab32649d39b829bc7679e6b28"
format_stamp: "Formatted at 2026-09-28 02:38:15 on dist-test-slave-fz47"
server_key: "acf1318994c8e48e51f95008fbffd8d5"
server_key_iv: "56193148049bd48bef2d00c68f606cc6"
server_key_version: "encryptionkey@0"
I20260928 02:38:15.409548 19726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:15.416285 19726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:15.417973 19726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:15.418084 19726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:15.418278 19726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:38:15.418555 19726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:38:15.418586 19726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:15.418615 19726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:38:15.418637 19726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:15.423888 19726 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41341
I20260928 02:38:15.423944 19854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41341 every 8 connection(s)
I20260928 02:38:15.424201 19726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
I20260928 02:38:15.426491 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 19726
I20260928 02:38:15.426622 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-1/wal/instance
I20260928 02:38:15.426816 392 external_mini_cluster.cc:1428] Setting key 86db1ba3bee2cea47bd37a22d1d5f2ff
Sep 28 02:38:15 dist-test-slave-fz47 krb5kdc[19485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:38:15.433743 19569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:15.424805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39861 (local address 127.0.98.62:35781)
0928 02:38:15.424844 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:15.424846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:15.425046 (+ 200us) server_negotiation.cc:407] Connection header received
0928 02:38:15.425997 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:15.425998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:15.426011 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:15.426060 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:15.426788 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.427372 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.428682 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.428952 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.429021 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:15.431221 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:15.431222 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:15.431222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:15.431228 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:15.432411 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:15.432842 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:15.432842 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:15.432843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:15.432906 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:15.433160 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:15.433160 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:15.433160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:15.433248 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:15.433326 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:15.433579 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 02:38:15.433656 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:38:15.433893 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:15.424775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35781 (local address 127.0.98.2:39861)
0928 02:38:15.424937 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:15.424969 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:15.425898 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:15.426124 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:15.426128 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:15.426419 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:15.426690 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:15.426696 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.427443 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.427444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:15.428602 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:15.428604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.429030 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.429031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:15.429126 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:15.429593 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:15.429601 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:15.431134 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:15.432487 (+ 1353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:15.432494 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:15.432497 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:15.432774 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:15.432969 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:15.432969 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:15.432970 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:15.433100 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:15.433319 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:15.433326 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:15.433464 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 02:38:15.433550 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 02:38:15.433691 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":87,"threads_started":1}
I20260928 02:38:15.434319 19855 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35781
I20260928 02:38:15.434415 19855 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 02:38:15.434594 19855 heartbeater.cc:511] Master 127.0.98.62:35781 requested a full tablet report, sending...
I20260928 02:38:15.434919 19530 ts_manager.cc:194] Registered new tserver with Master: 7f176d6ab32649d39b829bc7679e6b28 (127.0.98.2:41341)
I20260928 02:38:15.435240 19530 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 02:38:15.449057 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35781
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:15.525429 19860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:15.525622 19860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:15.525650 19860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:15.526762 19860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:15.526800 19860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:15.526834 19860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:38:15.528250 19860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:15.528473 19860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:15.528671 19860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:15.533478 19866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:15.533497 19868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:15.533528 19860 server_base.cc:1034] running on GCE node
W20260928 02:38:15.533527 19865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:15.533840 19860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:15.534060 19860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:15.535190 19860 hybrid_clock.cc:648] HybridClock initialized: now 1790563095535178 us; error 27 us; skew 500 ppm
Sep 28 02:38:15 dist-test-slave-fz47 krb5kdc[19485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:15.537668 19860 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 02:38:15.538059 19860 webserver.cc:460] Webserver started at http://127.0.98.3:35125/ using document root <none> and password file <none>
I20260928 02:38:15.538257 19860 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:15.538301 19860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:15.538447 19860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:15.539424 19860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-2/data/instance:
uuid: "3cf1fd22cb6a4389957c547095cd8088"
format_stamp: "Formatted at 2026-09-28 02:38:15 on dist-test-slave-fz47"
server_key: "8892760cabbfea22f6b168b37a372c8b"
server_key_iv: "71079c59f7bc9e1c493594ca371f0ead"
server_key_version: "encryptionkey@0"
I20260928 02:38:15.539723 19860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3cf1fd22cb6a4389957c547095cd8088"
format_stamp: "Formatted at 2026-09-28 02:38:15 on dist-test-slave-fz47"
server_key: "8892760cabbfea22f6b168b37a372c8b"
server_key_iv: "71079c59f7bc9e1c493594ca371f0ead"
server_key_version: "encryptionkey@0"
I20260928 02:38:15.540885 19860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:38:15.541508 19875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:15.541648 19860 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:15.541723 19860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "3cf1fd22cb6a4389957c547095cd8088"
format_stamp: "Formatted at 2026-09-28 02:38:15 on dist-test-slave-fz47"
server_key: "8892760cabbfea22f6b168b37a372c8b"
server_key_iv: "71079c59f7bc9e1c493594ca371f0ead"
server_key_version: "encryptionkey@0"
I20260928 02:38:15.541778 19860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:15.552664 19860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:15.554598 19860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:15.554704 19860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:15.554864 19860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:38:15.555099 19860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:38:15.555127 19860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:15.555145 19860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:38:15.555163 19860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:15.560312 19860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40735
I20260928 02:38:15.560370 19988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40735 every 8 connection(s)
I20260928 02:38:15.560644 19860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
I20260928 02:38:15.562930 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 19860
I20260928 02:38:15.563028 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-2/wal/instance
I20260928 02:38:15.563186 392 external_mini_cluster.cc:1428] Setting key a2b85c268195c008dc9b4299501d06a1
Sep 28 02:38:15 dist-test-slave-fz47 krb5kdc[19485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:38:15.569224 19569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:15.561305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56355 (local address 127.0.98.62:35781)
0928 02:38:15.561369 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:15.561371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:15.561558 (+ 187us) server_negotiation.cc:407] Connection header received
0928 02:38:15.562498 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:15.562499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:15.562509 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:15.562557 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:15.563223 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.563884 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.564929 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.565121 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.565171 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:15.567136 (+ 1965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:15.567137 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:15.567138 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:15.567144 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:15.568206 (+ 1062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:15.568553 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:15.568554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:15.568554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:15.568601 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:15.568769 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:15.568770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:15.568770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:15.568860 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:15.568901 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:15.569103 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 02:38:15.569160 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 02:38:15.569422 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:15.561295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35781 (local address 127.0.98.3:56355)
0928 02:38:15.561435 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:15.561474 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:15.562407 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:15.562622 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:15.562626 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:15.562881 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:15.563140 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:15.563146 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.563969 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.563971 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:15.564860 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:15.564864 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.565180 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.565180 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:15.565297 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:15.565724 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:15.565732 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:15.567059 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:15.568268 (+ 1209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:15.568274 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:15.568276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:15.568497 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:15.568645 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:15.568646 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:15.568646 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:15.568729 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:15.568899 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:15.568901 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:15.569012 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 02:38:15.569086 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 02:38:15.569199 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
I20260928 02:38:15.569856 19989 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35781
I20260928 02:38:15.569938 19989 heartbeater.cc:464] Registering TS with master...
I20260928 02:38:15.570122 19989 heartbeater.cc:511] Master 127.0.98.62:35781 requested a full tablet report, sending...
I20260928 02:38:15.570453 19530 ts_manager.cc:194] Registered new tserver with Master: 3cf1fd22cb6a4389957c547095cd8088 (127.0.98.3:40735)
I20260928 02:38:15.570781 19530 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56355
I20260928 02:38:15.574688 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:38:15.581153 19569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:15.575874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50104 (local address 127.0.98.62:35781)
0928 02:38:15.575910 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:15.575912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:15.576036 (+ 124us) server_negotiation.cc:407] Connection header received
0928 02:38:15.576092 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:15.576093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:15.576103 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:15.576147 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:15.576721 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.577184 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.577912 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.578091 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.578143 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:15.579204 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:15.579205 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:15.579205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:15.579211 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:15.580180 (+ 969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:15.580474 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:15.580474 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:15.580475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:15.580523 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:15.580675 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:15.580676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:15.580677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:15.580808 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:15.580899 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:15.580996 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 02:38:15.581065 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:38:15.582564 19530 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:38:15.584081 19530 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:38:15.590206 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:15.586063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40735 (local address 127.0.0.1:43764)
0928 02:38:15.586875 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:15.586885 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:15.586936 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:15.587133 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:15.587134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:15.587136 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:15.587175 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:15.587176 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.588197 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.588198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:15.589093 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:15.589095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.589944 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.589945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:15.590002 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:15.590013 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 02:38:15.590083 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 02:38:15.590131 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"thread_start_us":80,"threads_started":1}
I20260928 02:38:15.590224 20001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:15.586302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43764 (local address 127.0.98.3:40735)
0928 02:38:15.586485 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:15.586490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:15.586988 (+ 498us) server_negotiation.cc:407] Connection header received
0928 02:38:15.587028 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:15.587030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:15.587045 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:15.587112 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:15.587231 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.588115 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.589163 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.589871 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.589947 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:15.590008 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:15.590102 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 02:38:15.590143 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":120,"threads_started":1}
I20260928 02:38:15.590274 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:15.586051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36921 (local address 127.0.0.1:50616)
0928 02:38:15.586210 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:15.586246 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:15.586310 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:15.586566 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:15.586569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:15.586571 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:15.586630 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:15.586631 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.587872 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.587872 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:15.589196 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:15.589199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.590048 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.590049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:15.590100 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:15.590103 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:38:15.590144 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 02:38:15.590175 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":30,"thread_start_us":55,"threads_started":1}
I20260928 02:38:15.590464 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:15.586240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50616 (local address 127.0.98.1:36921)
0928 02:38:15.586394 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:15.586404 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:15.586415 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:38:15.586457 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:15.586460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:15.586476 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:15.586563 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:15.586677 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.587777 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.589302 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.589980 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.590042 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:15.590091 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:15.590344 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 02:38:15.590391 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":70,"threads_started":1}
I20260928 02:38:15.590987 19923 tablet_service.cc:1467] Processing CreateTablet for tablet e8809590bdf54473aae8790bea3fcf4d (DEFAULT_TABLE table=test-table [id=540c9cc3b89b4c8799609b34e072926e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:15.591096 19655 tablet_service.cc:1467] Processing CreateTablet for tablet e8809590bdf54473aae8790bea3fcf4d (DEFAULT_TABLE table=test-table [id=540c9cc3b89b4c8799609b34e072926e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:15.591295 19923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8809590bdf54473aae8790bea3fcf4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:15.591428 19655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8809590bdf54473aae8790bea3fcf4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:15.591693 19998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:15.586051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41341 (local address 127.0.0.1:47958)
0928 02:38:15.586491 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:15.586500 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:15.586542 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:15.586753 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:15.586754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:15.586755 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:15.586797 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:15.586798 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.587767 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.587768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:15.589203 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:15.589204 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.591393 (+ 2189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.591394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:15.591455 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:15.591459 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:38:15.591591 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 02:38:15.591628 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":48,"thread_start_us":59,"threads_started":1}
I20260928 02:38:15.591724 20002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:15.586401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47958 (local address 127.0.98.2:41341)
0928 02:38:15.586595 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:15.586598 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:15.586607 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:38:15.586643 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:15.586645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:15.586658 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:15.586754 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:15.586871 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.587682 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.590125 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.591297 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.591366 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:15.591420 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:15.591607 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 02:38:15.591649 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":86,"threads_started":1}
I20260928 02:38:15.592612 19789 tablet_service.cc:1467] Processing CreateTablet for tablet e8809590bdf54473aae8790bea3fcf4d (DEFAULT_TABLE table=test-table [id=540c9cc3b89b4c8799609b34e072926e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:15.592919 19789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8809590bdf54473aae8790bea3fcf4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:15.594094 20003 tablet_bootstrap.cc:492] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088: Bootstrap starting.
I20260928 02:38:15.594488 20004 tablet_bootstrap.cc:492] T e8809590bdf54473aae8790bea3fcf4d P df84da30e48449828e76c728190ea9f8: Bootstrap starting.
I20260928 02:38:15.594750 20003 tablet_bootstrap.cc:654] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:15.595109 20004 tablet_bootstrap.cc:654] T e8809590bdf54473aae8790bea3fcf4d P df84da30e48449828e76c728190ea9f8: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:15.595293 20003 log.cc:824] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:15.595610 20004 log.cc:824] T e8809590bdf54473aae8790bea3fcf4d P df84da30e48449828e76c728190ea9f8: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:15.596037 20003 tablet_bootstrap.cc:492] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088: No bootstrap required, opened a new log
I20260928 02:38:15.596082 20006 tablet_bootstrap.cc:492] T e8809590bdf54473aae8790bea3fcf4d P 7f176d6ab32649d39b829bc7679e6b28: Bootstrap starting.
I20260928 02:38:15.596122 20003 ts_tablet_manager.cc:1397] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:38:15.596262 20004 tablet_bootstrap.cc:492] T e8809590bdf54473aae8790bea3fcf4d P df84da30e48449828e76c728190ea9f8: No bootstrap required, opened a new log
I20260928 02:38:15.596308 20004 ts_tablet_manager.cc:1397] T e8809590bdf54473aae8790bea3fcf4d P df84da30e48449828e76c728190ea9f8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:38:15.596655 20006 tablet_bootstrap.cc:654] T e8809590bdf54473aae8790bea3fcf4d P 7f176d6ab32649d39b829bc7679e6b28: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:15.597070 20006 log.cc:824] T e8809590bdf54473aae8790bea3fcf4d P 7f176d6ab32649d39b829bc7679e6b28: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:15.597677 20006 tablet_bootstrap.cc:492] T e8809590bdf54473aae8790bea3fcf4d P 7f176d6ab32649d39b829bc7679e6b28: No bootstrap required, opened a new log
I20260928 02:38:15.597734 20006 ts_tablet_manager.cc:1397] T e8809590bdf54473aae8790bea3fcf4d P 7f176d6ab32649d39b829bc7679e6b28: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:38:15.598033 20003 raft_consensus.cc:348] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df84da30e48449828e76c728190ea9f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 } } peers { permanent_uuid: "7f176d6ab32649d39b829bc7679e6b28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41341 } } peers { permanent_uuid: "3cf1fd22cb6a4389957c547095cd8088" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40735 } }
I20260928 02:38:15.598212 20003 raft_consensus.cc:374] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:15.598124 20004 raft_consensus.cc:348] T e8809590bdf54473aae8790bea3fcf4d P df84da30e48449828e76c728190ea9f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df84da30e48449828e76c728190ea9f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 } } peers { permanent_uuid: "7f176d6ab32649d39b829bc7679e6b28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41341 } } peers { permanent_uuid: "3cf1fd22cb6a4389957c547095cd8088" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40735 } }
I20260928 02:38:15.598256 20004 raft_consensus.cc:374] T e8809590bdf54473aae8790bea3fcf4d P df84da30e48449828e76c728190ea9f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:15.598255 20003 raft_consensus.cc:729] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cf1fd22cb6a4389957c547095cd8088, State: Initialized, Role: FOLLOWER
I20260928 02:38:15.598285 20004 raft_consensus.cc:729] T e8809590bdf54473aae8790bea3fcf4d P df84da30e48449828e76c728190ea9f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df84da30e48449828e76c728190ea9f8, State: Initialized, Role: FOLLOWER
I20260928 02:38:15.598369 20003 consensus_queue.cc:260] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df84da30e48449828e76c728190ea9f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 } } peers { permanent_uuid: "7f176d6ab32649d39b829bc7679e6b28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41341 } } peers { permanent_uuid: "3cf1fd22cb6a4389957c547095cd8088" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40735 } }
I20260928 02:38:15.598400 20004 consensus_queue.cc:260] T e8809590bdf54473aae8790bea3fcf4d P df84da30e48449828e76c728190ea9f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df84da30e48449828e76c728190ea9f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 } } peers { permanent_uuid: "7f176d6ab32649d39b829bc7679e6b28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41341 } } peers { permanent_uuid: "3cf1fd22cb6a4389957c547095cd8088" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40735 } }
I20260928 02:38:15.598642 20004 ts_tablet_manager.cc:1428] T e8809590bdf54473aae8790bea3fcf4d P df84da30e48449828e76c728190ea9f8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:38:15.598642 20003 ts_tablet_manager.cc:1428] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:38:15.598784 19721 heartbeater.cc:503] Master 127.0.98.62:35781 was elected leader, sending a full tablet report...
I20260928 02:38:15.599041 19989 heartbeater.cc:503] Master 127.0.98.62:35781 was elected leader, sending a full tablet report...
I20260928 02:38:15.599444 20006 raft_consensus.cc:348] T e8809590bdf54473aae8790bea3fcf4d P 7f176d6ab32649d39b829bc7679e6b28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df84da30e48449828e76c728190ea9f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 } } peers { permanent_uuid: "7f176d6ab32649d39b829bc7679e6b28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41341 } } peers { permanent_uuid: "3cf1fd22cb6a4389957c547095cd8088" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40735 } }
I20260928 02:38:15.599599 20006 raft_consensus.cc:374] T e8809590bdf54473aae8790bea3fcf4d P 7f176d6ab32649d39b829bc7679e6b28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:15.599634 20006 raft_consensus.cc:729] T e8809590bdf54473aae8790bea3fcf4d P 7f176d6ab32649d39b829bc7679e6b28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f176d6ab32649d39b829bc7679e6b28, State: Initialized, Role: FOLLOWER
I20260928 02:38:15.599730 20006 consensus_queue.cc:260] T e8809590bdf54473aae8790bea3fcf4d P 7f176d6ab32649d39b829bc7679e6b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df84da30e48449828e76c728190ea9f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 } } peers { permanent_uuid: "7f176d6ab32649d39b829bc7679e6b28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41341 } } peers { permanent_uuid: "3cf1fd22cb6a4389957c547095cd8088" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40735 } }
I20260928 02:38:15.599989 20006 ts_tablet_manager.cc:1428] T e8809590bdf54473aae8790bea3fcf4d P 7f176d6ab32649d39b829bc7679e6b28: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:38:15.600085 19855 heartbeater.cc:503] Master 127.0.98.62:35781 was elected leader, sending a full tablet report...
I20260928 02:38:15.660806 20009 raft_consensus.cc:482] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:38:15.660936 20009 raft_consensus.cc:504] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df84da30e48449828e76c728190ea9f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 } } peers { permanent_uuid: "7f176d6ab32649d39b829bc7679e6b28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41341 } } peers { permanent_uuid: "3cf1fd22cb6a4389957c547095cd8088" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40735 } }
I20260928 02:38:15.661309 20009 leader_election.cc:288] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df84da30e48449828e76c728190ea9f8 (127.0.98.1:36921), 7f176d6ab32649d39b829bc7679e6b28 (127.0.98.2:41341)
I20260928 02:38:15.664958 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:15.661566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54395 (local address 127.0.98.1:36921)
0928 02:38:15.661616 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:15.661618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:15.661626 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:38:15.661653 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:15.661654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:15.661667 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:15.661718 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:15.661863 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.662616 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.664046 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.664609 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.664680 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:15.664690 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:15.664838 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 02:38:15.664905 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 02:38:15.664939 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:15.661425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36921 (local address 127.0.98.3:54395)
0928 02:38:15.661484 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:15.661496 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:15.661541 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:15.661731 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:15.661732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:15.661734 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:15.661800 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:15.661802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.662682 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.662682 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:15.663972 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:15.663974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.664678 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.664679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:15.664773 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:15.664776 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:38:15.664823 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 02:38:15.664865 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":76}
I20260928 02:38:15.664986 20002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:15.661589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58627 (local address 127.0.98.2:41341)
0928 02:38:15.661650 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:15.661652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:15.661660 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:38:15.661714 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:15.661715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:15.661723 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:15.661769 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:15.661891 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.662603 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.664041 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.664608 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.664706 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:15.664714 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:15.664890 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 02:38:15.664928 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:38:15.665141 20012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:15.661466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41341 (local address 127.0.98.3:58627)
0928 02:38:15.661607 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:15.661616 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:15.661660 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:15.661775 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:15.661775 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:15.661777 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:15.661829 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:15.661830 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.662675 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.662675 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:15.663964 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:15.663967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.664744 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.664745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:15.664824 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:15.664827 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:38:15.664873 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 02:38:15.664912 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":2,"thread_start_us":78,"threads_started":1}
I20260928 02:38:15.665246 19675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8809590bdf54473aae8790bea3fcf4d" candidate_uuid: "3cf1fd22cb6a4389957c547095cd8088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df84da30e48449828e76c728190ea9f8" is_pre_election: true
I20260928 02:38:15.665421 19675 raft_consensus.cc:2454] T e8809590bdf54473aae8790bea3fcf4d P df84da30e48449828e76c728190ea9f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cf1fd22cb6a4389957c547095cd8088 in term 0.
I20260928 02:38:15.665521 19809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8809590bdf54473aae8790bea3fcf4d" candidate_uuid: "3cf1fd22cb6a4389957c547095cd8088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f176d6ab32649d39b829bc7679e6b28" is_pre_election: true
I20260928 02:38:15.665632 19876 leader_election.cc:302] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cf1fd22cb6a4389957c547095cd8088, df84da30e48449828e76c728190ea9f8; no voters:
I20260928 02:38:15.665668 19809 raft_consensus.cc:2454] T e8809590bdf54473aae8790bea3fcf4d P 7f176d6ab32649d39b829bc7679e6b28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cf1fd22cb6a4389957c547095cd8088 in term 0.
I20260928 02:38:15.665755 20009 raft_consensus.cc:2781] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:38:15.665800 20009 raft_consensus.cc:482] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:38:15.665833 20009 raft_consensus.cc:3037] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:15.666513 20009 raft_consensus.cc:504] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df84da30e48449828e76c728190ea9f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 } } peers { permanent_uuid: "7f176d6ab32649d39b829bc7679e6b28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41341 } } peers { permanent_uuid: "3cf1fd22cb6a4389957c547095cd8088" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40735 } }
I20260928 02:38:15.666633 20009 leader_election.cc:288] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088 [CANDIDATE]: Term 1 election: Requested vote from peers df84da30e48449828e76c728190ea9f8 (127.0.98.1:36921), 7f176d6ab32649d39b829bc7679e6b28 (127.0.98.2:41341)
I20260928 02:38:15.666811 19675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8809590bdf54473aae8790bea3fcf4d" candidate_uuid: "3cf1fd22cb6a4389957c547095cd8088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df84da30e48449828e76c728190ea9f8"
I20260928 02:38:15.666839 19809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8809590bdf54473aae8790bea3fcf4d" candidate_uuid: "3cf1fd22cb6a4389957c547095cd8088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f176d6ab32649d39b829bc7679e6b28"
I20260928 02:38:15.666899 19809 raft_consensus.cc:3037] T e8809590bdf54473aae8790bea3fcf4d P 7f176d6ab32649d39b829bc7679e6b28 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:15.666898 19675 raft_consensus.cc:3037] T e8809590bdf54473aae8790bea3fcf4d P df84da30e48449828e76c728190ea9f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:15.667572 19809 raft_consensus.cc:2454] T e8809590bdf54473aae8790bea3fcf4d P 7f176d6ab32649d39b829bc7679e6b28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cf1fd22cb6a4389957c547095cd8088 in term 1.
I20260928 02:38:15.667701 19675 raft_consensus.cc:2454] T e8809590bdf54473aae8790bea3fcf4d P df84da30e48449828e76c728190ea9f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cf1fd22cb6a4389957c547095cd8088 in term 1.
I20260928 02:38:15.667735 19878 leader_election.cc:302] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cf1fd22cb6a4389957c547095cd8088, 7f176d6ab32649d39b829bc7679e6b28; no voters:
I20260928 02:38:15.667812 20009 raft_consensus.cc:2781] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:15.667960 20009 raft_consensus.cc:686] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088 [term 1 LEADER]: Becoming Leader. State: Replica: 3cf1fd22cb6a4389957c547095cd8088, State: Running, Role: LEADER
I20260928 02:38:15.668054 20009 consensus_queue.cc:237] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df84da30e48449828e76c728190ea9f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 } } peers { permanent_uuid: "7f176d6ab32649d39b829bc7679e6b28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41341 } } peers { permanent_uuid: "3cf1fd22cb6a4389957c547095cd8088" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40735 } }
I20260928 02:38:15.668864 19529 catalog_manager.cc:5495] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cf1fd22cb6a4389957c547095cd8088 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3cf1fd22cb6a4389957c547095cd8088" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df84da30e48449828e76c728190ea9f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f176d6ab32649d39b829bc7679e6b28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cf1fd22cb6a4389957c547095cd8088" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40735 } health_report { overall_health: HEALTHY } } }
W20260928 02:38:15.674671 19856 tablet.cc:2369] T e8809590bdf54473aae8790bea3fcf4d P 7f176d6ab32649d39b829bc7679e6b28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:38:15.682492 20001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:15.680424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43772 (local address 127.0.98.3:40735)
0928 02:38:15.680476 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:15.680478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:15.680486 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:38:15.680518 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:15.680519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:15.680535 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:15.680595 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:38:15.680694 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.681145 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.681888 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.682082 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.682133 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:15.682230 (+ 97us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:38:15.682320 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:38:15.682373 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:15.682406 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 02:38:15.682422 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 02:38:15.684056 19809 raft_consensus.cc:1261] T e8809590bdf54473aae8790bea3fcf4d P 7f176d6ab32649d39b829bc7679e6b28 [term 1 FOLLOWER]: Refusing update from remote peer 3cf1fd22cb6a4389957c547095cd8088: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:38:15.684056 19675 raft_consensus.cc:1261] T e8809590bdf54473aae8790bea3fcf4d P df84da30e48449828e76c728190ea9f8 [term 1 FOLLOWER]: Refusing update from remote peer 3cf1fd22cb6a4389957c547095cd8088: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:38:15.684383 20013 consensus_queue.cc:1035] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f176d6ab32649d39b829bc7679e6b28" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:38:15.684508 20009 consensus_queue.cc:1035] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df84da30e48449828e76c728190ea9f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:38:15.686710 20016 mvcc.cc:205] Tried to move back new op lower bound from 7334146439919554560 to 7334146439857094656. Current Snapshot: MvccSnapshot[applied={T|T < 7334146439919554560}]
I20260928 02:38:15.687475 20017 mvcc.cc:205] Tried to move back new op lower bound from 7334146439919554560 to 7334146439857094656. Current Snapshot: MvccSnapshot[applied={T|T < 7334146439919554560}]
I20260928 02:38:15.689323 19530 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:38:15.689404 19530 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:38:15.690049 19530 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92fc23b21ce44360aee72d658f7450e6: Sending DeleteTablet for 3 replicas of tablet e8809590bdf54473aae8790bea3fcf4d
I20260928 02:38:15.690377 19923 tablet_service.cc:1514] Processing DeleteTablet for tablet e8809590bdf54473aae8790bea3fcf4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43764
I20260928 02:38:15.690385 19655 tablet_service.cc:1514] Processing DeleteTablet for tablet e8809590bdf54473aae8790bea3fcf4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50616
I20260928 02:38:15.690577 19789 tablet_service.cc:1514] Processing DeleteTablet for tablet e8809590bdf54473aae8790bea3fcf4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47958
I20260928 02:38:15.690763 20027 tablet_replica.cc:330] T e8809590bdf54473aae8790bea3fcf4d P 7f176d6ab32649d39b829bc7679e6b28: stopping tablet replica
I20260928 02:38:15.690836 20025 tablet_replica.cc:330] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088: stopping tablet replica
I20260928 02:38:15.690929 20025 raft_consensus.cc:2229] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:38:15.691035 20025 raft_consensus.cc:2258] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:38:15.691025 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 19592
I20260928 02:38:15.691709 20025 ts_tablet_manager.cc:1905] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:38:15.693331 20025 ts_tablet_manager.cc:1918] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:38:15.693409 20025 log.cc:1196] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-2/wal/wals/e8809590bdf54473aae8790bea3fcf4d
I20260928 02:38:15.693656 20025 ts_tablet_manager.cc:1939] T e8809590bdf54473aae8790bea3fcf4d P 3cf1fd22cb6a4389957c547095cd8088: Deleting consensus metadata
I20260928 02:38:15.693815 20027 raft_consensus.cc:2229] T e8809590bdf54473aae8790bea3fcf4d P 7f176d6ab32649d39b829bc7679e6b28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:38:15.693949 20027 raft_consensus.cc:2258] T e8809590bdf54473aae8790bea3fcf4d P 7f176d6ab32649d39b829bc7679e6b28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:38:15.694042 19517 catalog_manager.cc:4841] TS 3cf1fd22cb6a4389957c547095cd8088 (127.0.98.3:40735): tablet e8809590bdf54473aae8790bea3fcf4d (table test-table [id=540c9cc3b89b4c8799609b34e072926e]) successfully deleted
I20260928 02:38:15.694298 20027 ts_tablet_manager.cc:1905] T e8809590bdf54473aae8790bea3fcf4d P 7f176d6ab32649d39b829bc7679e6b28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:38:15.695367 20027 ts_tablet_manager.cc:1918] T e8809590bdf54473aae8790bea3fcf4d P 7f176d6ab32649d39b829bc7679e6b28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:38:15.695425 20027 log.cc:1196] T e8809590bdf54473aae8790bea3fcf4d P 7f176d6ab32649d39b829bc7679e6b28: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790562862804661-392-0/minicluster-data/ts-1/wal/wals/e8809590bdf54473aae8790bea3fcf4d
I20260928 02:38:15.695619 20027 ts_tablet_manager.cc:1939] T e8809590bdf54473aae8790bea3fcf4d P 7f176d6ab32649d39b829bc7679e6b28: Deleting consensus metadata
I20260928 02:38:15.695977 19516 catalog_manager.cc:4841] TS 7f176d6ab32649d39b829bc7679e6b28 (127.0.98.2:41341): tablet e8809590bdf54473aae8790bea3fcf4d (table test-table [id=540c9cc3b89b4c8799609b34e072926e]) successfully deleted
W20260928 02:38:15.699805 19514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36921 (error 108)
I20260928 02:38:15.700232 19998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:15.700112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36921 (local address 127.0.0.1:50620)
0928 02:38:15.700163 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:15.700199 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 02:38:15.700266 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 19726
W20260928 02:38:15.700887 19514 catalog_manager.cc:4584] TS df84da30e48449828e76c728190ea9f8 (127.0.98.1:36921): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e8809590bdf54473aae8790bea3fcf4d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36921: connect: Connection refused (error 111)
I20260928 02:38:15.703812 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 19860
I20260928 02:38:15.707504 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 19498
2026-09-28T02:38:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1786 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T02:38:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:38:15Z Disabled control of system clock
I20260928 02:38:15.731779 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43689
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:38:15.808636 20032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:15.808854 20032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:15.808874 20032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:15.809985 20032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:15.810019 20032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:15.810030 20032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:15.810042 20032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:15.810056 20032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:15.811316 20032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43689
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:15.811499 20032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:15.811724 20032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:15.816102 20037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:15.816116 20040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:15.816116 20038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:15.816239 20032 server_base.cc:1034] running on GCE node
I20260928 02:38:15.816550 20032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:15.816814 20032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:15.817951 20032 hybrid_clock.cc:648] HybridClock initialized: now 1790563095817936 us; error 34 us; skew 500 ppm
I20260928 02:38:15.819065 20032 webserver.cc:460] Webserver started at http://127.0.98.62:33987/ using document root <none> and password file <none>
I20260928 02:38:15.819262 20032 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:15.819307 20032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:15.819424 20032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:15.820411 20032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "72f6cdc86b2045f78635918682ae4a64"
format_stamp: "Formatted at 2026-09-28 02:38:15 on dist-test-slave-fz47"
server_key: "6f609b562c492d05b9cfcaaafbdca94c"
server_key_iv: "473e52f96d71d456d6720ab6bb029df6"
server_key_version: "encryptionkey@0"
I20260928 02:38:15.820727 20032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "72f6cdc86b2045f78635918682ae4a64"
format_stamp: "Formatted at 2026-09-28 02:38:15 on dist-test-slave-fz47"
server_key: "6f609b562c492d05b9cfcaaafbdca94c"
server_key_iv: "473e52f96d71d456d6720ab6bb029df6"
server_key_version: "encryptionkey@0"
I20260928 02:38:15.821974 20032 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:38:15.822595 20046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:15.822738 20032 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:15.822817 20032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "72f6cdc86b2045f78635918682ae4a64"
format_stamp: "Formatted at 2026-09-28 02:38:15 on dist-test-slave-fz47"
server_key: "6f609b562c492d05b9cfcaaafbdca94c"
server_key_iv: "473e52f96d71d456d6720ab6bb029df6"
server_key_version: "encryptionkey@0"
I20260928 02:38:15.822875 20032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:15.832687 20032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:15.832957 20032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:15.833077 20032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:15.836586 20032 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43689
I20260928 02:38:15.836619 20098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43689 every 8 connection(s)
I20260928 02:38:15.836908 20032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:38:15.837824 20099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:15.839923 20099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72f6cdc86b2045f78635918682ae4a64: Bootstrap starting.
I20260928 02:38:15.840411 20099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72f6cdc86b2045f78635918682ae4a64: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:15.840644 20099 log.cc:824] T 00000000000000000000000000000000 P 72f6cdc86b2045f78635918682ae4a64: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:15.841199 20099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72f6cdc86b2045f78635918682ae4a64: No bootstrap required, opened a new log
I20260928 02:38:15.842692 20099 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72f6cdc86b2045f78635918682ae4a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72f6cdc86b2045f78635918682ae4a64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43689 } }
I20260928 02:38:15.842823 20099 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72f6cdc86b2045f78635918682ae4a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:15.842868 20099 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72f6cdc86b2045f78635918682ae4a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72f6cdc86b2045f78635918682ae4a64, State: Initialized, Role: FOLLOWER
I20260928 02:38:15.842967 20099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72f6cdc86b2045f78635918682ae4a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72f6cdc86b2045f78635918682ae4a64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43689 } }
I20260928 02:38:15.843016 20099 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72f6cdc86b2045f78635918682ae4a64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:38:15.843034 20099 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72f6cdc86b2045f78635918682ae4a64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:38:15.843065 20099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72f6cdc86b2045f78635918682ae4a64 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:15.843478 20099 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72f6cdc86b2045f78635918682ae4a64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72f6cdc86b2045f78635918682ae4a64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43689 } }
I20260928 02:38:15.843554 20099 leader_election.cc:302] T 00000000000000000000000000000000 P 72f6cdc86b2045f78635918682ae4a64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72f6cdc86b2045f78635918682ae4a64; no voters:
I20260928 02:38:15.843665 20099 leader_election.cc:288] T 00000000000000000000000000000000 P 72f6cdc86b2045f78635918682ae4a64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:38:15.843719 20102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72f6cdc86b2045f78635918682ae4a64 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:15.843865 20102 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72f6cdc86b2045f78635918682ae4a64 [term 1 LEADER]: Becoming Leader. State: Replica: 72f6cdc86b2045f78635918682ae4a64, State: Running, Role: LEADER
I20260928 02:38:15.843885 20099 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72f6cdc86b2045f78635918682ae4a64 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:38:15.843961 20102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72f6cdc86b2045f78635918682ae4a64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72f6cdc86b2045f78635918682ae4a64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43689 } }
I20260928 02:38:15.844374 20104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72f6cdc86b2045f78635918682ae4a64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72f6cdc86b2045f78635918682ae4a64. Latest consensus state: current_term: 1 leader_uuid: "72f6cdc86b2045f78635918682ae4a64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72f6cdc86b2045f78635918682ae4a64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43689 } } }
I20260928 02:38:15.844491 20103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72f6cdc86b2045f78635918682ae4a64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72f6cdc86b2045f78635918682ae4a64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72f6cdc86b2045f78635918682ae4a64" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43689 } } }
I20260928 02:38:15.844548 20104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72f6cdc86b2045f78635918682ae4a64 [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:15.844564 20103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72f6cdc86b2045f78635918682ae4a64 [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:15.844825 20111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:38:15.845316 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 20032
I20260928 02:38:15.845337 20111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:38:15.845422 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:38:15.845597 392 external_mini_cluster.cc:1428] Setting key 454ab17c0663072f93e5e080d1f68366
I20260928 02:38:15.846840 20111 catalog_manager.cc:1347] Generated new cluster ID: 3725cef8a33e4aa5a47e243a8c01509d
I20260928 02:38:15.846889 20111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:38:15.851392 20121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:15.846007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52370 (local address 127.0.98.62:43689)
0928 02:38:15.846166 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:15.846174 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:15.846355 (+ 181us) server_negotiation.cc:407] Connection header received
0928 02:38:15.846503 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:15.846511 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:15.846578 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:15.846711 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:15.846972 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.848306 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.849381 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.850009 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.850125 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:15.850721 (+ 596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:15.850725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:15.850725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:15.850936 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:15.850976 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:15.850990 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:15.851051 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:15.851116 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 02:38:15.851191 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":62,"threads_started":1}
I20260928 02:38:15.862406 20111 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:38:15.863066 20111 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:38:15.869789 20111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72f6cdc86b2045f78635918682ae4a64: Generated new TSK 0
I20260928 02:38:15.869962 20111 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:38:15.871835 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43689
--builtin_ntp_servers=127.0.98.20:34901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:38:15.950090 20123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:15.950296 20123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:15.950325 20123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:15.951429 20123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:15.951465 20123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:15.951498 20123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 02:38:15.952782 20123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:15.952996 20123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:15.953275 20123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:38:15.958752 20123 server_base.cc:1034] running on GCE node
W20260928 02:38:15.958712 20131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:15.958710 20129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:15.958860 20128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:15.959131 20123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:15.959360 20123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:15.960506 20123 hybrid_clock.cc:648] HybridClock initialized: now 1790563095960492 us; error 29 us; skew 500 ppm
I20260928 02:38:15.961679 20123 webserver.cc:460] Webserver started at http://127.0.98.1:42385/ using document root <none> and password file <none>
I20260928 02:38:15.961874 20123 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:15.961921 20123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:15.962039 20123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:15.963011 20123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-0/data/instance:
uuid: "1ba70c864ae145688fc120b9fb66db2f"
format_stamp: "Formatted at 2026-09-28 02:38:15 on dist-test-slave-fz47"
server_key: "3eb8ea407076b79b35b7ef488d6ee32e"
server_key_iv: "25a74efa91994dae900c57f46b72732f"
server_key_version: "encryptionkey@0"
I20260928 02:38:15.963315 20123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1ba70c864ae145688fc120b9fb66db2f"
format_stamp: "Formatted at 2026-09-28 02:38:15 on dist-test-slave-fz47"
server_key: "3eb8ea407076b79b35b7ef488d6ee32e"
server_key_iv: "25a74efa91994dae900c57f46b72732f"
server_key_version: "encryptionkey@0"
I20260928 02:38:15.964452 20123 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:38:15.965087 20137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:15.965250 20123 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:15.965363 20123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-0/wal
uuid: "1ba70c864ae145688fc120b9fb66db2f"
format_stamp: "Formatted at 2026-09-28 02:38:15 on dist-test-slave-fz47"
server_key: "3eb8ea407076b79b35b7ef488d6ee32e"
server_key_iv: "25a74efa91994dae900c57f46b72732f"
server_key_version: "encryptionkey@0"
I20260928 02:38:15.965428 20123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:15.974956 20123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:15.975188 20123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:15.975278 20123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:15.975432 20123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:38:15.975682 20123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:38:15.975710 20123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:15.975729 20123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:38:15.975744 20123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:15.980870 20123 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39651
I20260928 02:38:15.980955 20250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39651 every 8 connection(s)
I20260928 02:38:15.981207 20123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-0/data/info.pb
I20260928 02:38:15.986007 20121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:15.981891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60587 (local address 127.0.98.62:43689)
0928 02:38:15.981934 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:15.981936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:15.982150 (+ 214us) server_negotiation.cc:407] Connection header received
0928 02:38:15.982640 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:15.982642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:15.982657 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:15.982711 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:15.983362 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.983849 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.984788 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.984974 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.985027 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:15.985749 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:15.985751 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:15.985751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:15.985756 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:15.985771 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:15.985775 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:15.985823 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:15.985899 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 02:38:15.985930 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:38:15.986197 20253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:15.981840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43689 (local address 127.0.98.1:60587)
0928 02:38:15.982031 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:15.982067 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:15.982546 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:15.982790 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:15.982794 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:15.982803 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:15.983265 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:15.983270 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.983935 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.983936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:15.984704 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:15.984706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:15.985045 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:15.985046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:15.985172 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:15.985629 (+ 457us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:38:15.985629 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:15.985655 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:38:15.985676 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:15.985828 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:15.985834 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:15.985837 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:38:15.985899 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 02:38:15.985996 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
I20260928 02:38:15.986330 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 20123
I20260928 02:38:15.986438 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-0/wal/instance
I20260928 02:38:15.986626 392 external_mini_cluster.cc:1428] Setting key 1492c06a5a5c9db11f9dc562a744c904
I20260928 02:38:15.986652 20251 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43689
I20260928 02:38:15.986739 20251 heartbeater.cc:464] Registering TS with master...
I20260928 02:38:15.986923 20251 heartbeater.cc:511] Master 127.0.98.62:43689 requested a full tablet report, sending...
I20260928 02:38:15.987457 20063 ts_manager.cc:194] Registered new tserver with Master: 1ba70c864ae145688fc120b9fb66db2f (127.0.98.1:39651)
I20260928 02:38:15.987489 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43689
--builtin_ntp_servers=127.0.98.20:34901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:38:15.988050 20063 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:60587
W20260928 02:38:16.064998 20254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:16.065274 20254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:16.065307 20254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:16.066454 20254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:16.066491 20254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:16.066515 20254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 02:38:16.067828 20254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:16.068030 20254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:16.068287 20254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:16.072980 20259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:16.073024 20254 server_base.cc:1034] running on GCE node
W20260928 02:38:16.073084 20260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:16.073206 20262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:16.073457 20254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:16.073665 20254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:16.074826 20254 hybrid_clock.cc:648] HybridClock initialized: now 1790563096074807 us; error 32 us; skew 500 ppm
I20260928 02:38:16.075990 20254 webserver.cc:460] Webserver started at http://127.0.98.2:42837/ using document root <none> and password file <none>
I20260928 02:38:16.076196 20254 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:16.076247 20254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:16.076372 20254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:16.077502 20254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-1/data/instance:
uuid: "08c0a0a600ad4a2a8eae384d435611e9"
format_stamp: "Formatted at 2026-09-28 02:38:16 on dist-test-slave-fz47"
server_key: "de7b909fa6af674bd53d8415208fc0f2"
server_key_iv: "d98263705f36d6ec7372bc7fe42293f0"
server_key_version: "encryptionkey@0"
I20260928 02:38:16.077826 20254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-1/wal/instance:
uuid: "08c0a0a600ad4a2a8eae384d435611e9"
format_stamp: "Formatted at 2026-09-28 02:38:16 on dist-test-slave-fz47"
server_key: "de7b909fa6af674bd53d8415208fc0f2"
server_key_iv: "d98263705f36d6ec7372bc7fe42293f0"
server_key_version: "encryptionkey@0"
I20260928 02:38:16.078975 20254 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:38:16.079654 20268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:16.079804 20254 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:16.079880 20254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-1/wal
uuid: "08c0a0a600ad4a2a8eae384d435611e9"
format_stamp: "Formatted at 2026-09-28 02:38:16 on dist-test-slave-fz47"
server_key: "de7b909fa6af674bd53d8415208fc0f2"
server_key_iv: "d98263705f36d6ec7372bc7fe42293f0"
server_key_version: "encryptionkey@0"
I20260928 02:38:16.079943 20254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:16.089004 20254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:16.089323 20254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:16.089450 20254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:16.089638 20254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:38:16.089946 20254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:38:16.089977 20254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:16.090003 20254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:38:16.090027 20254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:16.095438 20254 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42763
I20260928 02:38:16.095475 20381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42763 every 8 connection(s)
I20260928 02:38:16.095764 20254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-1/data/info.pb
I20260928 02:38:16.100565 20121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:16.096460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49035 (local address 127.0.98.62:43689)
0928 02:38:16.096522 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:16.096525 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:16.096700 (+ 175us) server_negotiation.cc:407] Connection header received
0928 02:38:16.097176 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:16.097177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:16.097186 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:16.097295 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:16.097929 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:16.098424 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:16.099350 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:16.099523 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:16.099566 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:16.100364 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:16.100365 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:16.100366 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:16.100370 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:16.100382 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:16.100385 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:16.100424 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:16.100487 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 02:38:16.100510 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 02:38:16.100745 20384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:16.096428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43689 (local address 127.0.98.2:49035)
0928 02:38:16.096594 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:16.096626 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:16.097087 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:16.097363 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:16.097366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:16.097374 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:16.097837 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:16.097840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:16.098506 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:16.098507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:16.099270 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:16.099272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:16.099586 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:16.099587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:16.099694 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:16.100183 (+ 489us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:38:16.100183 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:16.100211 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:38:16.100273 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:16.100435 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:16.100437 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:16.100440 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:38:16.100483 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 02:38:16.100555 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
I20260928 02:38:16.101141 20382 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43689
I20260928 02:38:16.101282 20382 heartbeater.cc:464] Registering TS with master...
I20260928 02:38:16.101424 20382 heartbeater.cc:511] Master 127.0.98.62:43689 requested a full tablet report, sending...
I20260928 02:38:16.101671 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 20254
I20260928 02:38:16.101758 20063 ts_manager.cc:194] Registered new tserver with Master: 08c0a0a600ad4a2a8eae384d435611e9 (127.0.98.2:42763)
I20260928 02:38:16.101784 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-1/wal/instance
I20260928 02:38:16.101948 392 external_mini_cluster.cc:1428] Setting key f451bab58c854d61ff17ae3f0aa5ead8
I20260928 02:38:16.102149 20063 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:49035
I20260928 02:38:16.102897 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43689
--builtin_ntp_servers=127.0.98.20:34901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:38:16.182329 20385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:16.182533 20385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:16.182555 20385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:16.183691 20385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:16.183727 20385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:16.183753 20385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 02:38:16.185026 20385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:16.185197 20385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:16.185454 20385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:16.189919 20391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:16.189930 20390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:16.189929 20393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:16.190017 20385 server_base.cc:1034] running on GCE node
I20260928 02:38:16.190255 20385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:16.190486 20385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:16.191633 20385 hybrid_clock.cc:648] HybridClock initialized: now 1790563096191618 us; error 33 us; skew 500 ppm
I20260928 02:38:16.192736 20385 webserver.cc:460] Webserver started at http://127.0.98.3:41277/ using document root <none> and password file <none>
I20260928 02:38:16.192966 20385 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:16.193014 20385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:16.193141 20385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:16.194180 20385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-2/data/instance:
uuid: "78a611eb3a8d4a4cb2bd4383d4d7403b"
format_stamp: "Formatted at 2026-09-28 02:38:16 on dist-test-slave-fz47"
server_key: "42c23690f92b8cadc2e8c3d3e4b1e3c8"
server_key_iv: "24c6d2b92ca4f829da081b6e49f56137"
server_key_version: "encryptionkey@0"
I20260928 02:38:16.194523 20385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-2/wal/instance:
uuid: "78a611eb3a8d4a4cb2bd4383d4d7403b"
format_stamp: "Formatted at 2026-09-28 02:38:16 on dist-test-slave-fz47"
server_key: "42c23690f92b8cadc2e8c3d3e4b1e3c8"
server_key_iv: "24c6d2b92ca4f829da081b6e49f56137"
server_key_version: "encryptionkey@0"
I20260928 02:38:16.195631 20385 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:38:16.196233 20399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:16.196382 20385 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:16.196451 20385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-2/wal
uuid: "78a611eb3a8d4a4cb2bd4383d4d7403b"
format_stamp: "Formatted at 2026-09-28 02:38:16 on dist-test-slave-fz47"
server_key: "42c23690f92b8cadc2e8c3d3e4b1e3c8"
server_key_iv: "24c6d2b92ca4f829da081b6e49f56137"
server_key_version: "encryptionkey@0"
I20260928 02:38:16.196516 20385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:16.208606 20385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:16.208864 20385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:16.208979 20385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:16.209172 20385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:38:16.209512 20385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:38:16.209547 20385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:16.209574 20385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:38:16.209591 20385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:16.214586 20385 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36031
I20260928 02:38:16.214668 20512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36031 every 8 connection(s)
I20260928 02:38:16.214919 20385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-2/data/info.pb
I20260928 02:38:16.217090 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 20385
I20260928 02:38:16.217183 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-2/wal/instance
I20260928 02:38:16.217386 392 external_mini_cluster.cc:1428] Setting key 68e81cbad301a687e8c2e9f9ce9bc9e2
I20260928 02:38:16.219882 20121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:16.215528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56601 (local address 127.0.98.62:43689)
0928 02:38:16.215571 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:16.215573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:16.215765 (+ 192us) server_negotiation.cc:407] Connection header received
0928 02:38:16.216234 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:16.216235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:16.216245 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:16.216291 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:16.216906 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:16.217547 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:16.218693 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:16.218896 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:16.218947 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:16.219674 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:16.219675 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:16.219676 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:16.219680 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:16.219692 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:16.219696 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:16.219734 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:16.219787 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 02:38:16.219824 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:38:16.220037 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:16.215510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43689 (local address 127.0.98.3:56601)
0928 02:38:16.215665 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:16.215697 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:16.216153 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:16.216354 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:16.216357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:16.216365 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:16.216844 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:16.216849 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:16.217628 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:16.217630 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:16.218620 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:16.218623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:16.218954 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:16.218954 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:16.219053 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:16.219535 (+ 482us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:38:16.219535 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:16.219573 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:38:16.219599 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:16.219736 (+ 137us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:16.219739 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:16.219741 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 02:38:16.219784 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 02:38:16.219859 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
I20260928 02:38:16.220434 20513 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43689
I20260928 02:38:16.220508 20513 heartbeater.cc:464] Registering TS with master...
I20260928 02:38:16.220643 20513 heartbeater.cc:511] Master 127.0.98.62:43689 requested a full tablet report, sending...
I20260928 02:38:16.220937 20062 ts_manager.cc:194] Registered new tserver with Master: 78a611eb3a8d4a4cb2bd4383d4d7403b (127.0.98.3:36031)
I20260928 02:38:16.221333 20062 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:56601
I20260928 02:38:16.228870 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:38:16.232846 20121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:16.230140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52372 (local address 127.0.98.62:43689)
0928 02:38:16.230210 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:16.230211 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:16.230248 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:38:16.230321 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:16.230322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:16.230333 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:16.230380 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:16.230538 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:16.230997 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:16.231791 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:16.231972 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:16.232024 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:16.232625 (+ 601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:16.232627 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:16.232627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:16.232631 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:16.232644 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:16.232649 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:16.232688 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:16.232750 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 02:38:16.232773 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:38:16.234145 20062 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:38:16.235554 20062 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:38:16.241328 20521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:16.237341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36031 (local address 127.0.0.1:50844)
0928 02:38:16.237447 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:16.237486 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:16.237544 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:16.237954 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:16.237956 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:16.237959 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:16.238016 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:16.238018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:16.239279 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:16.239280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:16.240413 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:16.240415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:16.241028 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:16.241029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:16.241105 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:16.241117 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 02:38:16.241186 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 02:38:16.241235 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":43,"thread_start_us":58,"threads_started":1}
I20260928 02:38:16.241338 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:16.237658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50844 (local address 127.0.98.3:36031)
0928 02:38:16.237814 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:16.237817 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:16.237825 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:38:16.237858 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:16.237860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:16.237874 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:16.237950 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:16.238078 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:16.239200 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:16.240481 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:16.240954 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:16.241016 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:16.241104 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:16.241200 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 02:38:16.241234 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
I20260928 02:38:16.241906 20522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:16.237532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39651 (local address 127.0.0.1:40802)
0928 02:38:16.237825 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:16.237834 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:16.237887 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:16.238279 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:16.238279 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:16.238281 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:16.238316 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:16.238317 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:16.239132 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:16.239133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:16.240426 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:16.240428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:16.241672 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:16.241673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:16.241743 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:16.241747 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:38:16.241799 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 02:38:16.241836 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":25,"thread_start_us":63,"threads_started":1}
I20260928 02:38:16.241952 20447 tablet_service.cc:1467] Processing CreateTablet for tablet 1e8b53736da140c9a883a56fdbe52c8f (DEFAULT_TABLE table=test-table [id=6e9a6a0f8c1e4e1885223cd63bed1863]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:16.242204 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:16.237627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40802 (local address 127.0.98.1:39651)
0928 02:38:16.237787 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:16.237792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:16.237890 (+ 98us) server_negotiation.cc:407] Connection header received
0928 02:38:16.238130 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:16.238134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:16.238149 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:16.238214 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:16.238371 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:16.239059 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:16.240519 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:16.241244 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:16.241347 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:16.242045 (+ 698us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:16.242073 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 02:38:16.242128 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20260928 02:38:16.242311 20447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e8b53736da140c9a883a56fdbe52c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:16.242461 20524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:16.237377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42763 (local address 127.0.0.1:41224)
0928 02:38:16.237664 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:16.237675 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:16.237724 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:16.237995 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:16.237995 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:16.237997 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:16.238066 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:16.238069 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:16.240188 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:16.240189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:16.241200 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:16.241203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:16.242267 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:16.242268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:16.242327 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:16.242331 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:38:16.242367 (+ 36us) client_negotiation.cc:240] Negotiation successful
0928 02:38:16.242401 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":75,"threads_started":1}
I20260928 02:38:16.242833 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:16.237486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41224 (local address 127.0.98.2:42763)
0928 02:38:16.237832 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:16.237838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:16.237848 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:38:16.237886 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:16.237888 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:16.237902 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:16.237995 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:16.238141 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:16.239209 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:16.241347 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:16.242161 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:16.242604 (+ 443us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:16.242696 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:16.242720 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 02:38:16.242759 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":87,"threads_started":1}
I20260928 02:38:16.242825 20185 tablet_service.cc:1467] Processing CreateTablet for tablet 1e8b53736da140c9a883a56fdbe52c8f (DEFAULT_TABLE table=test-table [id=6e9a6a0f8c1e4e1885223cd63bed1863]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:16.243129 20185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e8b53736da140c9a883a56fdbe52c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:16.245456 20316 tablet_service.cc:1467] Processing CreateTablet for tablet 1e8b53736da140c9a883a56fdbe52c8f (DEFAULT_TABLE table=test-table [id=6e9a6a0f8c1e4e1885223cd63bed1863]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:38:16.245699 20528 tablet_bootstrap.cc:492] T 1e8b53736da140c9a883a56fdbe52c8f P 1ba70c864ae145688fc120b9fb66db2f: Bootstrap starting.
I20260928 02:38:16.245733 20316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e8b53736da140c9a883a56fdbe52c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:16.246327 20528 tablet_bootstrap.cc:654] T 1e8b53736da140c9a883a56fdbe52c8f P 1ba70c864ae145688fc120b9fb66db2f: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:16.246637 20528 log.cc:824] T 1e8b53736da140c9a883a56fdbe52c8f P 1ba70c864ae145688fc120b9fb66db2f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:16.247339 20528 tablet_bootstrap.cc:492] T 1e8b53736da140c9a883a56fdbe52c8f P 1ba70c864ae145688fc120b9fb66db2f: No bootstrap required, opened a new log
I20260928 02:38:16.247408 20528 ts_tablet_manager.cc:1397] T 1e8b53736da140c9a883a56fdbe52c8f P 1ba70c864ae145688fc120b9fb66db2f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:38:16.248795 20530 tablet_bootstrap.cc:492] T 1e8b53736da140c9a883a56fdbe52c8f P 08c0a0a600ad4a2a8eae384d435611e9: Bootstrap starting.
I20260928 02:38:16.248965 20527 tablet_bootstrap.cc:492] T 1e8b53736da140c9a883a56fdbe52c8f P 78a611eb3a8d4a4cb2bd4383d4d7403b: Bootstrap starting.
I20260928 02:38:16.249389 20530 tablet_bootstrap.cc:654] T 1e8b53736da140c9a883a56fdbe52c8f P 08c0a0a600ad4a2a8eae384d435611e9: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:16.249608 20527 tablet_bootstrap.cc:654] T 1e8b53736da140c9a883a56fdbe52c8f P 78a611eb3a8d4a4cb2bd4383d4d7403b: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:16.249639 20530 log.cc:824] T 1e8b53736da140c9a883a56fdbe52c8f P 08c0a0a600ad4a2a8eae384d435611e9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:16.249895 20527 log.cc:824] T 1e8b53736da140c9a883a56fdbe52c8f P 78a611eb3a8d4a4cb2bd4383d4d7403b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:16.249867 20528 raft_consensus.cc:348] T 1e8b53736da140c9a883a56fdbe52c8f P 1ba70c864ae145688fc120b9fb66db2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a611eb3a8d4a4cb2bd4383d4d7403b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36031 } } peers { permanent_uuid: "1ba70c864ae145688fc120b9fb66db2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39651 } } peers { permanent_uuid: "08c0a0a600ad4a2a8eae384d435611e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42763 } }
I20260928 02:38:16.250038 20528 raft_consensus.cc:374] T 1e8b53736da140c9a883a56fdbe52c8f P 1ba70c864ae145688fc120b9fb66db2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:16.250080 20528 raft_consensus.cc:729] T 1e8b53736da140c9a883a56fdbe52c8f P 1ba70c864ae145688fc120b9fb66db2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ba70c864ae145688fc120b9fb66db2f, State: Initialized, Role: FOLLOWER
I20260928 02:38:16.250188 20528 consensus_queue.cc:260] T 1e8b53736da140c9a883a56fdbe52c8f P 1ba70c864ae145688fc120b9fb66db2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a611eb3a8d4a4cb2bd4383d4d7403b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36031 } } peers { permanent_uuid: "1ba70c864ae145688fc120b9fb66db2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39651 } } peers { permanent_uuid: "08c0a0a600ad4a2a8eae384d435611e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42763 } }
I20260928 02:38:16.250273 20530 tablet_bootstrap.cc:492] T 1e8b53736da140c9a883a56fdbe52c8f P 08c0a0a600ad4a2a8eae384d435611e9: No bootstrap required, opened a new log
I20260928 02:38:16.250337 20530 ts_tablet_manager.cc:1397] T 1e8b53736da140c9a883a56fdbe52c8f P 08c0a0a600ad4a2a8eae384d435611e9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 02:38:16.250416 20528 ts_tablet_manager.cc:1428] T 1e8b53736da140c9a883a56fdbe52c8f P 1ba70c864ae145688fc120b9fb66db2f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 02:38:16.250483 20251 heartbeater.cc:503] Master 127.0.98.62:43689 was elected leader, sending a full tablet report...
I20260928 02:38:16.250993 20527 tablet_bootstrap.cc:492] T 1e8b53736da140c9a883a56fdbe52c8f P 78a611eb3a8d4a4cb2bd4383d4d7403b: No bootstrap required, opened a new log
I20260928 02:38:16.251039 20527 ts_tablet_manager.cc:1397] T 1e8b53736da140c9a883a56fdbe52c8f P 78a611eb3a8d4a4cb2bd4383d4d7403b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 02:38:16.252362 20530 raft_consensus.cc:348] T 1e8b53736da140c9a883a56fdbe52c8f P 08c0a0a600ad4a2a8eae384d435611e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a611eb3a8d4a4cb2bd4383d4d7403b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36031 } } peers { permanent_uuid: "1ba70c864ae145688fc120b9fb66db2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39651 } } peers { permanent_uuid: "08c0a0a600ad4a2a8eae384d435611e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42763 } }
I20260928 02:38:16.252480 20530 raft_consensus.cc:374] T 1e8b53736da140c9a883a56fdbe52c8f P 08c0a0a600ad4a2a8eae384d435611e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:16.252508 20530 raft_consensus.cc:729] T 1e8b53736da140c9a883a56fdbe52c8f P 08c0a0a600ad4a2a8eae384d435611e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08c0a0a600ad4a2a8eae384d435611e9, State: Initialized, Role: FOLLOWER
I20260928 02:38:16.252614 20530 consensus_queue.cc:260] T 1e8b53736da140c9a883a56fdbe52c8f P 08c0a0a600ad4a2a8eae384d435611e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a611eb3a8d4a4cb2bd4383d4d7403b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36031 } } peers { permanent_uuid: "1ba70c864ae145688fc120b9fb66db2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39651 } } peers { permanent_uuid: "08c0a0a600ad4a2a8eae384d435611e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42763 } }
I20260928 02:38:16.252853 20530 ts_tablet_manager.cc:1428] T 1e8b53736da140c9a883a56fdbe52c8f P 08c0a0a600ad4a2a8eae384d435611e9: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 02:38:16.252889 20382 heartbeater.cc:503] Master 127.0.98.62:43689 was elected leader, sending a full tablet report...
I20260928 02:38:16.252933 20527 raft_consensus.cc:348] T 1e8b53736da140c9a883a56fdbe52c8f P 78a611eb3a8d4a4cb2bd4383d4d7403b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a611eb3a8d4a4cb2bd4383d4d7403b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36031 } } peers { permanent_uuid: "1ba70c864ae145688fc120b9fb66db2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39651 } } peers { permanent_uuid: "08c0a0a600ad4a2a8eae384d435611e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42763 } }
I20260928 02:38:16.253062 20527 raft_consensus.cc:374] T 1e8b53736da140c9a883a56fdbe52c8f P 78a611eb3a8d4a4cb2bd4383d4d7403b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:16.253091 20527 raft_consensus.cc:729] T 1e8b53736da140c9a883a56fdbe52c8f P 78a611eb3a8d4a4cb2bd4383d4d7403b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78a611eb3a8d4a4cb2bd4383d4d7403b, State: Initialized, Role: FOLLOWER
I20260928 02:38:16.253209 20527 consensus_queue.cc:260] T 1e8b53736da140c9a883a56fdbe52c8f P 78a611eb3a8d4a4cb2bd4383d4d7403b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a611eb3a8d4a4cb2bd4383d4d7403b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36031 } } peers { permanent_uuid: "1ba70c864ae145688fc120b9fb66db2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39651 } } peers { permanent_uuid: "08c0a0a600ad4a2a8eae384d435611e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42763 } }
I20260928 02:38:16.253475 20527 ts_tablet_manager.cc:1428] T 1e8b53736da140c9a883a56fdbe52c8f P 78a611eb3a8d4a4cb2bd4383d4d7403b: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260928 02:38:16.253515 20513 heartbeater.cc:503] Master 127.0.98.62:43689 was elected leader, sending a full tablet report...
I20260928 02:38:16.322172 20533 raft_consensus.cc:482] T 1e8b53736da140c9a883a56fdbe52c8f P 1ba70c864ae145688fc120b9fb66db2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:38:16.322315 20533 raft_consensus.cc:504] T 1e8b53736da140c9a883a56fdbe52c8f P 1ba70c864ae145688fc120b9fb66db2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a611eb3a8d4a4cb2bd4383d4d7403b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36031 } } peers { permanent_uuid: "1ba70c864ae145688fc120b9fb66db2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39651 } } peers { permanent_uuid: "08c0a0a600ad4a2a8eae384d435611e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42763 } }
I20260928 02:38:16.322664 20533 leader_election.cc:288] T 1e8b53736da140c9a883a56fdbe52c8f P 1ba70c864ae145688fc120b9fb66db2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78a611eb3a8d4a4cb2bd4383d4d7403b (127.0.98.3:36031), 08c0a0a600ad4a2a8eae384d435611e9 (127.0.98.2:42763)
I20260928 02:38:16.326051 20253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:16.322821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42763 (local address 127.0.98.1:47891)
0928 02:38:16.322876 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:16.322887 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:16.322938 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:16.323093 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:16.323094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:16.323096 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:16.323167 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:16.323168 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:16.324068 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:16.324069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:16.325281 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:16.325284 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:16.325810 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:16.325811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:16.325875 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:16.325878 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 02:38:16.325933 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 02:38:16.325978 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":3}
I20260928 02:38:16.326051 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:16.322902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47891 (local address 127.0.98.2:42763)
0928 02:38:16.322945 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:16.322947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:16.322992 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:38:16.323020 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:16.323021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:16.323027 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:16.323069 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:16.323229 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:16.323996 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:16.325402 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:16.325737 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:16.325801 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:16.325810 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:16.325951 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 02:38:16.325988 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:38:16.326416 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:16.322918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42431 (local address 127.0.98.3:36031)
0928 02:38:16.322965 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:16.322966 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:16.323132 (+ 166us) server_negotiation.cc:407] Connection header received
0928 02:38:16.323173 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:16.323174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:16.323182 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:16.323284 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:38:16.323388 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:16.324051 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:16.325356 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:16.325733 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:16.325791 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:16.325800 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:16.326269 (+ 469us) server_negotiation.cc:299] Negotiation successful
0928 02:38:16.326348 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 02:38:16.326388 20336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8b53736da140c9a883a56fdbe52c8f" candidate_uuid: "1ba70c864ae145688fc120b9fb66db2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08c0a0a600ad4a2a8eae384d435611e9" is_pre_election: true
I20260928 02:38:16.326536 20336 raft_consensus.cc:2454] T 1e8b53736da140c9a883a56fdbe52c8f P 08c0a0a600ad4a2a8eae384d435611e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ba70c864ae145688fc120b9fb66db2f in term 0.
I20260928 02:38:16.326608 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:16.322821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36031 (local address 127.0.98.1:42431)
0928 02:38:16.323065 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:16.323073 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:16.323123 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:16.323283 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:16.323284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:16.323285 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:38:16.323332 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:16.323333 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:16.324110 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:16.324110 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:16.325289 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:16.325290 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:16.326080 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:16.326081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:16.326174 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:16.326179 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:38:16.326496 (+ 317us) client_negotiation.cc:240] Negotiation successful
0928 02:38:16.326537 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":77,"thread_start_us":105,"threads_started":1}
I20260928 02:38:16.326776 20139 leader_election.cc:302] T 1e8b53736da140c9a883a56fdbe52c8f P 1ba70c864ae145688fc120b9fb66db2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08c0a0a600ad4a2a8eae384d435611e9, 1ba70c864ae145688fc120b9fb66db2f; no voters:
I20260928 02:38:16.326936 20533 raft_consensus.cc:2781] T 1e8b53736da140c9a883a56fdbe52c8f P 1ba70c864ae145688fc120b9fb66db2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:38:16.326923 20467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8b53736da140c9a883a56fdbe52c8f" candidate_uuid: "1ba70c864ae145688fc120b9fb66db2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78a611eb3a8d4a4cb2bd4383d4d7403b" is_pre_election: true
I20260928 02:38:16.327000 20533 raft_consensus.cc:482] T 1e8b53736da140c9a883a56fdbe52c8f P 1ba70c864ae145688fc120b9fb66db2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:38:16.327023 20533 raft_consensus.cc:3037] T 1e8b53736da140c9a883a56fdbe52c8f P 1ba70c864ae145688fc120b9fb66db2f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:16.327059 20467 raft_consensus.cc:2454] T 1e8b53736da140c9a883a56fdbe52c8f P 78a611eb3a8d4a4cb2bd4383d4d7403b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ba70c864ae145688fc120b9fb66db2f in term 0.
I20260928 02:38:16.327696 20533 raft_consensus.cc:504] T 1e8b53736da140c9a883a56fdbe52c8f P 1ba70c864ae145688fc120b9fb66db2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a611eb3a8d4a4cb2bd4383d4d7403b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36031 } } peers { permanent_uuid: "1ba70c864ae145688fc120b9fb66db2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39651 } } peers { permanent_uuid: "08c0a0a600ad4a2a8eae384d435611e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42763 } }
I20260928 02:38:16.327805 20533 leader_election.cc:288] T 1e8b53736da140c9a883a56fdbe52c8f P 1ba70c864ae145688fc120b9fb66db2f [CANDIDATE]: Term 1 election: Requested vote from peers 78a611eb3a8d4a4cb2bd4383d4d7403b (127.0.98.3:36031), 08c0a0a600ad4a2a8eae384d435611e9 (127.0.98.2:42763)
I20260928 02:38:16.327939 20336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8b53736da140c9a883a56fdbe52c8f" candidate_uuid: "1ba70c864ae145688fc120b9fb66db2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08c0a0a600ad4a2a8eae384d435611e9"
I20260928 02:38:16.327956 20467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8b53736da140c9a883a56fdbe52c8f" candidate_uuid: "1ba70c864ae145688fc120b9fb66db2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78a611eb3a8d4a4cb2bd4383d4d7403b"
I20260928 02:38:16.328019 20467 raft_consensus.cc:3037] T 1e8b53736da140c9a883a56fdbe52c8f P 78a611eb3a8d4a4cb2bd4383d4d7403b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:16.328012 20336 raft_consensus.cc:3037] T 1e8b53736da140c9a883a56fdbe52c8f P 08c0a0a600ad4a2a8eae384d435611e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:16.328696 20467 raft_consensus.cc:2454] T 1e8b53736da140c9a883a56fdbe52c8f P 78a611eb3a8d4a4cb2bd4383d4d7403b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ba70c864ae145688fc120b9fb66db2f in term 1.
I20260928 02:38:16.328751 20336 raft_consensus.cc:2454] T 1e8b53736da140c9a883a56fdbe52c8f P 08c0a0a600ad4a2a8eae384d435611e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ba70c864ae145688fc120b9fb66db2f in term 1.
I20260928 02:38:16.328860 20140 leader_election.cc:302] T 1e8b53736da140c9a883a56fdbe52c8f P 1ba70c864ae145688fc120b9fb66db2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ba70c864ae145688fc120b9fb66db2f, 78a611eb3a8d4a4cb2bd4383d4d7403b; no voters:
I20260928 02:38:16.328987 20533 raft_consensus.cc:2781] T 1e8b53736da140c9a883a56fdbe52c8f P 1ba70c864ae145688fc120b9fb66db2f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:16.329109 20533 raft_consensus.cc:686] T 1e8b53736da140c9a883a56fdbe52c8f P 1ba70c864ae145688fc120b9fb66db2f [term 1 LEADER]: Becoming Leader. State: Replica: 1ba70c864ae145688fc120b9fb66db2f, State: Running, Role: LEADER
I20260928 02:38:16.329181 20533 consensus_queue.cc:237] T 1e8b53736da140c9a883a56fdbe52c8f P 1ba70c864ae145688fc120b9fb66db2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a611eb3a8d4a4cb2bd4383d4d7403b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36031 } } peers { permanent_uuid: "1ba70c864ae145688fc120b9fb66db2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39651 } } peers { permanent_uuid: "08c0a0a600ad4a2a8eae384d435611e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42763 } }
I20260928 02:38:16.330065 20062 catalog_manager.cc:5495] T 1e8b53736da140c9a883a56fdbe52c8f P 1ba70c864ae145688fc120b9fb66db2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ba70c864ae145688fc120b9fb66db2f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1ba70c864ae145688fc120b9fb66db2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a611eb3a8d4a4cb2bd4383d4d7403b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ba70c864ae145688fc120b9fb66db2f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08c0a0a600ad4a2a8eae384d435611e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42763 } health_report { overall_health: UNKNOWN } } }
W20260928 02:38:16.346243 20383 tablet.cc:2369] T 1e8b53736da140c9a883a56fdbe52c8f P 08c0a0a600ad4a2a8eae384d435611e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:38:16.355041 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:16.352844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40810 (local address 127.0.98.1:39651)
0928 02:38:16.352898 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:16.352899 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:16.352907 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:38:16.352941 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:16.352942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:16.352958 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:16.353012 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:38:16.353118 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:16.353619 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:16.354308 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:16.354517 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:16.354571 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:16.354698 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:38:16.354827 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:38:16.354902 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:16.354938 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 02:38:16.354955 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 02:38:16.356600 20336 raft_consensus.cc:1261] T 1e8b53736da140c9a883a56fdbe52c8f P 08c0a0a600ad4a2a8eae384d435611e9 [term 1 FOLLOWER]: Refusing update from remote peer 1ba70c864ae145688fc120b9fb66db2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:38:16.356600 20467 raft_consensus.cc:1261] T 1e8b53736da140c9a883a56fdbe52c8f P 78a611eb3a8d4a4cb2bd4383d4d7403b [term 1 FOLLOWER]: Refusing update from remote peer 1ba70c864ae145688fc120b9fb66db2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:38:16.356907 20533 consensus_queue.cc:1035] T 1e8b53736da140c9a883a56fdbe52c8f P 1ba70c864ae145688fc120b9fb66db2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "78a611eb3a8d4a4cb2bd4383d4d7403b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:38:16.357038 20537 consensus_queue.cc:1035] T 1e8b53736da140c9a883a56fdbe52c8f P 1ba70c864ae145688fc120b9fb66db2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08c0a0a600ad4a2a8eae384d435611e9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:38:16.359411 20542 mvcc.cc:205] Tried to move back new op lower bound from 7334146442674454528 to 7334146442565558272. Current Snapshot: MvccSnapshot[applied={T|T < 7334146442674454528}]
I20260928 02:38:16.360092 20541 mvcc.cc:205] Tried to move back new op lower bound from 7334146442674454528 to 7334146442565558272. Current Snapshot: MvccSnapshot[applied={T|T < 7334146442674454528 or (T in {7334146442674454528})}]
I20260928 02:38:16.361169 20063 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:38:16.361275 20063 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:38:16.362064 20063 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72f6cdc86b2045f78635918682ae4a64: Sending DeleteTablet for 3 replicas of tablet 1e8b53736da140c9a883a56fdbe52c8f
I20260928 02:38:16.362409 20185 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e8b53736da140c9a883a56fdbe52c8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:16 UTC) from {username='slave'} at 127.0.0.1:40802
I20260928 02:38:16.362615 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 20123
I20260928 02:38:16.364291 20447 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e8b53736da140c9a883a56fdbe52c8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:16 UTC) from {username='slave'} at 127.0.0.1:50844
I20260928 02:38:16.364485 20551 tablet_replica.cc:330] T 1e8b53736da140c9a883a56fdbe52c8f P 78a611eb3a8d4a4cb2bd4383d4d7403b: stopping tablet replica
I20260928 02:38:16.364583 20551 raft_consensus.cc:2229] T 1e8b53736da140c9a883a56fdbe52c8f P 78a611eb3a8d4a4cb2bd4383d4d7403b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:38:16.364674 20551 raft_consensus.cc:2258] T 1e8b53736da140c9a883a56fdbe52c8f P 78a611eb3a8d4a4cb2bd4383d4d7403b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:38:16.365041 20551 ts_tablet_manager.cc:1905] T 1e8b53736da140c9a883a56fdbe52c8f P 78a611eb3a8d4a4cb2bd4383d4d7403b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:38:16.365852 20316 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e8b53736da140c9a883a56fdbe52c8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:38:16 UTC) from {username='slave'} at 127.0.0.1:41224
I20260928 02:38:16.366420 20552 tablet_replica.cc:330] T 1e8b53736da140c9a883a56fdbe52c8f P 08c0a0a600ad4a2a8eae384d435611e9: stopping tablet replica
I20260928 02:38:16.366513 20552 raft_consensus.cc:2229] T 1e8b53736da140c9a883a56fdbe52c8f P 08c0a0a600ad4a2a8eae384d435611e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:38:16.366607 20552 raft_consensus.cc:2258] T 1e8b53736da140c9a883a56fdbe52c8f P 08c0a0a600ad4a2a8eae384d435611e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:38:16.366689 20551 ts_tablet_manager.cc:1918] T 1e8b53736da140c9a883a56fdbe52c8f P 78a611eb3a8d4a4cb2bd4383d4d7403b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:38:16.366753 20551 log.cc:1196] T 1e8b53736da140c9a883a56fdbe52c8f P 78a611eb3a8d4a4cb2bd4383d4d7403b: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-2/wal/wals/1e8b53736da140c9a883a56fdbe52c8f
I20260928 02:38:16.366967 20552 ts_tablet_manager.cc:1905] T 1e8b53736da140c9a883a56fdbe52c8f P 08c0a0a600ad4a2a8eae384d435611e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:38:16.366981 20551 ts_tablet_manager.cc:1939] T 1e8b53736da140c9a883a56fdbe52c8f P 78a611eb3a8d4a4cb2bd4383d4d7403b: Deleting consensus metadata
I20260928 02:38:16.367340 20049 catalog_manager.cc:4841] TS 78a611eb3a8d4a4cb2bd4383d4d7403b (127.0.98.3:36031): tablet 1e8b53736da140c9a883a56fdbe52c8f (table test-table [id=6e9a6a0f8c1e4e1885223cd63bed1863]) successfully deleted
I20260928 02:38:16.368146 20552 ts_tablet_manager.cc:1918] T 1e8b53736da140c9a883a56fdbe52c8f P 08c0a0a600ad4a2a8eae384d435611e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:38:16.368197 20552 log.cc:1196] T 1e8b53736da140c9a883a56fdbe52c8f P 08c0a0a600ad4a2a8eae384d435611e9: Deleting WAL directory at /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790562862804661-392-0/minicluster-data/ts-1/wal/wals/1e8b53736da140c9a883a56fdbe52c8f
I20260928 02:38:16.368397 20552 ts_tablet_manager.cc:1939] T 1e8b53736da140c9a883a56fdbe52c8f P 08c0a0a600ad4a2a8eae384d435611e9: Deleting consensus metadata
I20260928 02:38:16.368726 20048 catalog_manager.cc:4841] TS 08c0a0a600ad4a2a8eae384d435611e9 (127.0.98.2:42763): tablet 1e8b53736da140c9a883a56fdbe52c8f (table test-table [id=6e9a6a0f8c1e4e1885223cd63bed1863]) successfully deleted
W20260928 02:38:16.368993 20049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39651 (error 108)
I20260928 02:38:16.369194 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 20254
I20260928 02:38:16.369881 20524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:16.369768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39651 (local address 127.0.0.1:40814)
0928 02:38:16.369814 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:16.369853 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 02:38:16.370035 20049 catalog_manager.cc:4584] TS 1ba70c864ae145688fc120b9fb66db2f (127.0.98.1:39651): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1e8b53736da140c9a883a56fdbe52c8f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39651: connect: Connection refused (error 111)
I20260928 02:38:16.373111 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 20385
I20260928 02:38:16.377655 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 20032
2026-09-28T02:38:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (670 ms)
[----------] 6 tests from EncryptionPolicyTest (7806 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:38:16 dist-test-slave-fz47 krb5kdc[20556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:38:16 dist-test-slave-fz47 krb5kdc[20556](info): set up 2 sockets
Sep 28 02:38:16 dist-test-slave-fz47 krb5kdc[20556](info): commencing operation
krb5kdc: starting...
W20260928 02:38:17.320889 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:38:17 dist-test-slave-fz47 krb5kdc[20556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:38:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:38:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:38:17.429989 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37245
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:17.506613 20569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:17.506811 20569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:17.506841 20569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:17.507956 20569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:17.507997 20569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:17.508014 20569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:17.508030 20569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:17.508044 20569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:17.509519 20569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37245
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:17.509749 20569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:17.509956 20569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:17.514278 20577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:17.514307 20575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:17.514407 20574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:17.514590 20569 server_base.cc:1034] running on GCE node
I20260928 02:38:17.514791 20569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:17.515014 20569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:17.516147 20569 hybrid_clock.cc:648] HybridClock initialized: now 1790563097516132 us; error 30 us; skew 500 ppm
Sep 28 02:38:17 dist-test-slave-fz47 krb5kdc[20556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:17.518390 20569 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:38:17.518807 20569 webserver.cc:460] Webserver started at http://127.0.0.1:41487/ using document root <none> and password file <none>
I20260928 02:38:17.518987 20569 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:17.519032 20569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:17.519158 20569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:17.520118 20569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "adea47148b4a4b75a40c023f2d9731fc"
format_stamp: "Formatted at 2026-09-28 02:38:17 on dist-test-slave-fz47"
server_key: "ed9f252c6cad8397173ad0abf849cd6e"
server_key_iv: "6108ced01d7e2db273cf2c786352e8fb"
server_key_version: "encryptionkey@0"
I20260928 02:38:17.520427 20569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "adea47148b4a4b75a40c023f2d9731fc"
format_stamp: "Formatted at 2026-09-28 02:38:17 on dist-test-slave-fz47"
server_key: "ed9f252c6cad8397173ad0abf849cd6e"
server_key_iv: "6108ced01d7e2db273cf2c786352e8fb"
server_key_version: "encryptionkey@0"
I20260928 02:38:17.521620 20569 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:38:17.522204 20584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:17.522349 20569 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:17.522426 20569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "adea47148b4a4b75a40c023f2d9731fc"
format_stamp: "Formatted at 2026-09-28 02:38:17 on dist-test-slave-fz47"
server_key: "ed9f252c6cad8397173ad0abf849cd6e"
server_key_iv: "6108ced01d7e2db273cf2c786352e8fb"
server_key_version: "encryptionkey@0"
I20260928 02:38:17.522481 20569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:17.533330 20569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:17.535001 20569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:17.535128 20569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:17.538828 20569 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37245
I20260928 02:38:17.538861 20636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37245 every 8 connection(s)
I20260928 02:38:17.539165 20569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:38:17.539944 20637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:17.542196 20637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adea47148b4a4b75a40c023f2d9731fc: Bootstrap starting.
I20260928 02:38:17.542737 20637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adea47148b4a4b75a40c023f2d9731fc: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:17.542990 20637 log.cc:824] T 00000000000000000000000000000000 P adea47148b4a4b75a40c023f2d9731fc: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:17.543612 20637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adea47148b4a4b75a40c023f2d9731fc: No bootstrap required, opened a new log
I20260928 02:38:17.543982 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 20569
I20260928 02:38:17.544099 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:38:17.544273 392 external_mini_cluster.cc:1428] Setting key c7b50f064687a9bd3d10fa81d263e744
I20260928 02:38:17.545599 20637 raft_consensus.cc:348] T 00000000000000000000000000000000 P adea47148b4a4b75a40c023f2d9731fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adea47148b4a4b75a40c023f2d9731fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37245 } }
I20260928 02:38:17.545740 20637 raft_consensus.cc:374] T 00000000000000000000000000000000 P adea47148b4a4b75a40c023f2d9731fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:17.545773 20637 raft_consensus.cc:729] T 00000000000000000000000000000000 P adea47148b4a4b75a40c023f2d9731fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adea47148b4a4b75a40c023f2d9731fc, State: Initialized, Role: FOLLOWER
I20260928 02:38:17.545907 20637 consensus_queue.cc:260] T 00000000000000000000000000000000 P adea47148b4a4b75a40c023f2d9731fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adea47148b4a4b75a40c023f2d9731fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37245 } }
I20260928 02:38:17.545989 20637 raft_consensus.cc:388] T 00000000000000000000000000000000 P adea47148b4a4b75a40c023f2d9731fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:38:17.546023 20637 raft_consensus.cc:482] T 00000000000000000000000000000000 P adea47148b4a4b75a40c023f2d9731fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:38:17.546069 20637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adea47148b4a4b75a40c023f2d9731fc [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:17.546979 20637 raft_consensus.cc:504] T 00000000000000000000000000000000 P adea47148b4a4b75a40c023f2d9731fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adea47148b4a4b75a40c023f2d9731fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37245 } }
I20260928 02:38:17.547066 20637 leader_election.cc:302] T 00000000000000000000000000000000 P adea47148b4a4b75a40c023f2d9731fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adea47148b4a4b75a40c023f2d9731fc; no voters:
I20260928 02:38:17.547209 20637 leader_election.cc:288] T 00000000000000000000000000000000 P adea47148b4a4b75a40c023f2d9731fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:38:17.547243 20642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adea47148b4a4b75a40c023f2d9731fc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:17.547358 20642 raft_consensus.cc:686] T 00000000000000000000000000000000 P adea47148b4a4b75a40c023f2d9731fc [term 1 LEADER]: Becoming Leader. State: Replica: adea47148b4a4b75a40c023f2d9731fc, State: Running, Role: LEADER
I20260928 02:38:17.547452 20637 sys_catalog.cc:564] T 00000000000000000000000000000000 P adea47148b4a4b75a40c023f2d9731fc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:38:17.547433 20642 consensus_queue.cc:237] T 00000000000000000000000000000000 P adea47148b4a4b75a40c023f2d9731fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adea47148b4a4b75a40c023f2d9731fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37245 } }
I20260928 02:38:17.547726 20642 sys_catalog.cc:455] T 00000000000000000000000000000000 P adea47148b4a4b75a40c023f2d9731fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader adea47148b4a4b75a40c023f2d9731fc. Latest consensus state: current_term: 1 leader_uuid: "adea47148b4a4b75a40c023f2d9731fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adea47148b4a4b75a40c023f2d9731fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37245 } } }
I20260928 02:38:17.547791 20642 sys_catalog.cc:458] T 00000000000000000000000000000000 P adea47148b4a4b75a40c023f2d9731fc [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:17.548318 20644 sys_catalog.cc:455] T 00000000000000000000000000000000 P adea47148b4a4b75a40c023f2d9731fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adea47148b4a4b75a40c023f2d9731fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adea47148b4a4b75a40c023f2d9731fc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37245 } } }
I20260928 02:38:17.548401 20644 sys_catalog.cc:458] T 00000000000000000000000000000000 P adea47148b4a4b75a40c023f2d9731fc [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:17.548365 20646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:38:17.548959 20646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:38:17.550547 20646 catalog_manager.cc:1347] Generated new cluster ID: f5ea10646f684197a0299c94b512901c
I20260928 02:38:17.550598 20646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:38:17 dist-test-slave-fz47 krb5kdc[20556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:38:17.554653 20641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:17.544804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48411 (local address 127.0.0.1:37245)
0928 02:38:17.544952 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:17.544962 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:17.544987 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:38:17.545572 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:17.545578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:17.545624 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:17.545732 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:17.546487 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:17.547550 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:17.548580 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:17.549192 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:17.549325 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:17.551683 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:17.551687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:17.551687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:17.551698 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:17.553135 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:17.553545 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:17.553547 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:17.553548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:17.553644 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:17.553843 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:17.553844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:17.553844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:17.554067 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:17.554235 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:17.554375 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 02:38:17.554477 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":66,"threads_started":1}
I20260928 02:38:17.555769 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:38:17.559719 20646 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:38:17.560395 20646 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:38:17.562469 20641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:17.556950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33763 (local address 127.0.0.1:37245)
0928 02:38:17.556983 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:17.556984 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:17.557086 (+ 102us) server_negotiation.cc:407] Connection header received
0928 02:38:17.557145 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:17.557146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:17.557155 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:17.557194 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:17.557698 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:17.558170 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:17.558819 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:17.559041 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:17.559092 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:17.560209 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:17.560210 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:17.560210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:17.560226 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:17.561489 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:17.561802 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:17.561803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:17.561803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:17.561850 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:17.562026 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:17.562027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:17.562027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:17.562124 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:17.562205 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:17.562335 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 02:38:17.562392 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":17}
I20260928 02:38:17.562884 20646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adea47148b4a4b75a40c023f2d9731fc: Generated new TSK 0
I20260928 02:38:17.563020 20646 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:38:17.569967 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 20569
W20260928 02:38:17.573462 20554 connection.cc:535] client connection to 127.0.0.1:37245 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37245): Connection reset by peer (error 104)
2026-09-28T02:38:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1190 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:38:17 dist-test-slave-fz47 krb5kdc[20668](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:38:17 dist-test-slave-fz47 krb5kdc[20668](info): set up 2 sockets
Sep 28 02:38:17 dist-test-slave-fz47 krb5kdc[20668](info): commencing operation
krb5kdc: starting...
W20260928 02:38:18.516896 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:38:18 dist-test-slave-fz47 krb5kdc[20668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:38:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:38:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:38:18.624325 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36323
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:18.701298 20681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:18.701493 20681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:18.701522 20681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:18.702674 20681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:18.702716 20681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:18.702745 20681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:18.702761 20681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:18.702785 20681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:18.704252 20681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36323
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:18.704480 20681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:18.704684 20681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:18.709362 20687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:18.709462 20686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:18.709523 20681 server_base.cc:1034] running on GCE node
W20260928 02:38:18.709362 20689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:18.709770 20681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:18.710040 20681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:18.711199 20681 hybrid_clock.cc:648] HybridClock initialized: now 1790563098711171 us; error 41 us; skew 500 ppm
Sep 28 02:38:18 dist-test-slave-fz47 krb5kdc[20668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:18.717633 20681 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:38:18.718072 20681 webserver.cc:460] Webserver started at http://127.0.0.1:38157/ using document root <none> and password file <none>
I20260928 02:38:18.718297 20681 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:18.718346 20681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:18.718468 20681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:18.719502 20681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "b4e80692a3ef4a4db5a706197757323f"
format_stamp: "Formatted at 2026-09-28 02:38:18 on dist-test-slave-fz47"
server_key: "6e9e42b05fcc9b713c49b6c0a6acad98"
server_key_iv: "8353415ce6be74e8527795916020d221"
server_key_version: "encryptionkey@0"
I20260928 02:38:18.719828 20681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "b4e80692a3ef4a4db5a706197757323f"
format_stamp: "Formatted at 2026-09-28 02:38:18 on dist-test-slave-fz47"
server_key: "6e9e42b05fcc9b713c49b6c0a6acad98"
server_key_iv: "8353415ce6be74e8527795916020d221"
server_key_version: "encryptionkey@0"
I20260928 02:38:18.721026 20681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:38:18.721746 20696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:18.721890 20681 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:18.721977 20681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "b4e80692a3ef4a4db5a706197757323f"
format_stamp: "Formatted at 2026-09-28 02:38:18 on dist-test-slave-fz47"
server_key: "6e9e42b05fcc9b713c49b6c0a6acad98"
server_key_iv: "8353415ce6be74e8527795916020d221"
server_key_version: "encryptionkey@0"
I20260928 02:38:18.722043 20681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:18.746007 20681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:18.748132 20681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:18.748278 20681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:18.751982 20681 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36323
I20260928 02:38:18.752024 20748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36323 every 8 connection(s)
I20260928 02:38:18.752343 20681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:38:18.753116 20749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:18.755359 20749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4e80692a3ef4a4db5a706197757323f: Bootstrap starting.
I20260928 02:38:18.755905 20749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4e80692a3ef4a4db5a706197757323f: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:18.756148 20749 log.cc:824] T 00000000000000000000000000000000 P b4e80692a3ef4a4db5a706197757323f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:18.756794 20749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4e80692a3ef4a4db5a706197757323f: No bootstrap required, opened a new log
I20260928 02:38:18.758163 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 20681
I20260928 02:38:18.758285 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:38:18.758477 392 external_mini_cluster.cc:1428] Setting key 44b4689a75e6b15b16639cea8c8687b2
I20260928 02:38:18.758589 20749 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4e80692a3ef4a4db5a706197757323f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e80692a3ef4a4db5a706197757323f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36323 } }
I20260928 02:38:18.758736 20749 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4e80692a3ef4a4db5a706197757323f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:18.758769 20749 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4e80692a3ef4a4db5a706197757323f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4e80692a3ef4a4db5a706197757323f, State: Initialized, Role: FOLLOWER
I20260928 02:38:18.758911 20749 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4e80692a3ef4a4db5a706197757323f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e80692a3ef4a4db5a706197757323f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36323 } }
I20260928 02:38:18.758994 20749 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4e80692a3ef4a4db5a706197757323f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:38:18.759028 20749 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4e80692a3ef4a4db5a706197757323f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:38:18.759096 20749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4e80692a3ef4a4db5a706197757323f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:18.759904 20749 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4e80692a3ef4a4db5a706197757323f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e80692a3ef4a4db5a706197757323f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36323 } }
I20260928 02:38:18.759989 20749 leader_election.cc:302] T 00000000000000000000000000000000 P b4e80692a3ef4a4db5a706197757323f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4e80692a3ef4a4db5a706197757323f; no voters:
I20260928 02:38:18.760130 20749 leader_election.cc:288] T 00000000000000000000000000000000 P b4e80692a3ef4a4db5a706197757323f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:38:18.760169 20754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4e80692a3ef4a4db5a706197757323f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:18.760310 20754 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4e80692a3ef4a4db5a706197757323f [term 1 LEADER]: Becoming Leader. State: Replica: b4e80692a3ef4a4db5a706197757323f, State: Running, Role: LEADER
I20260928 02:38:18.760377 20749 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4e80692a3ef4a4db5a706197757323f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:38:18.760474 20754 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4e80692a3ef4a4db5a706197757323f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e80692a3ef4a4db5a706197757323f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36323 } }
I20260928 02:38:18.760819 20754 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4e80692a3ef4a4db5a706197757323f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4e80692a3ef4a4db5a706197757323f. Latest consensus state: current_term: 1 leader_uuid: "b4e80692a3ef4a4db5a706197757323f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e80692a3ef4a4db5a706197757323f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36323 } } }
I20260928 02:38:18.760828 20756 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4e80692a3ef4a4db5a706197757323f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4e80692a3ef4a4db5a706197757323f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e80692a3ef4a4db5a706197757323f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36323 } } }
I20260928 02:38:18.760977 20756 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4e80692a3ef4a4db5a706197757323f [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:18.761170 20754 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4e80692a3ef4a4db5a706197757323f [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:18.761690 20763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:38:18.762180 20763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:38:18.763643 20763 catalog_manager.cc:1347] Generated new cluster ID: 8e2acec27247493fa3fdc36c512fcebf
I20260928 02:38:18.763693 20763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:38:18.764056 20753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:18.758953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32993 (local address 127.0.0.1:36323)
0928 02:38:18.759190 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:18.759199 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:18.759230 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:38:18.759328 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:18.759333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:18.759380 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:18.759495 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:18.759729 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:18.760875 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:18.761937 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:18.762488 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:18.762612 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:18.763262 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:18.763266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:18.763266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:18.763478 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:18.763513 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:18.763528 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:18.763644 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:18.763821 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 02:38:18.763848 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":91,"threads_started":1}
I20260928 02:38:18.764912 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:38:18.769748 20753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:18.766457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43759 (local address 127.0.0.1:36323)
0928 02:38:18.766504 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:18.766506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:18.766555 (+ 49us) server_negotiation.cc:407] Connection header received
0928 02:38:18.766626 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:18.766627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:18.766637 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:18.766687 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:18.766799 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:18.767526 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:18.768465 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:18.768788 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:18.768852 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:18.769509 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:18.769510 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:18.769511 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:18.769515 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:18.769530 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:18.769534 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:18.769599 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:18.769663 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 02:38:18.769675 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 02:38:18.773326 20763 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:38:18.773805 20763 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:38:18.777427 20763 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4e80692a3ef4a4db5a706197757323f: Generated new TSK 0
I20260928 02:38:18.777588 20763 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:38:18.779696 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 20681
W20260928 02:38:18.783640 20666 connection.cc:535] client connection to 127.0.0.1:36323 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36323): Connection reset by peer (error 104)
2026-09-28T02:38:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1210 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:38:18 dist-test-slave-fz47 krb5kdc[20780](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:38:18 dist-test-slave-fz47 krb5kdc[20780](info): set up 2 sockets
Sep 28 02:38:18 dist-test-slave-fz47 krb5kdc[20780](info): commencing operation
krb5kdc: starting...
W20260928 02:38:19.720935 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:38:19 dist-test-slave-fz47 krb5kdc[20780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:38:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:38:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:38:19.825896 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35879
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:19.903024 20793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:19.903226 20793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:19.903257 20793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:19.904366 20793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:19.904402 20793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:19.904420 20793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:19.904436 20793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:19.904452 20793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:19.905874 20793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35879
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:19.906112 20793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:19.906304 20793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:19.910831 20801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:19.910858 20798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:19.910912 20793 server_base.cc:1034] running on GCE node
W20260928 02:38:19.910918 20799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:19.911253 20793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:19.911520 20793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:19.912676 20793 hybrid_clock.cc:648] HybridClock initialized: now 1790563099912662 us; error 28 us; skew 500 ppm
Sep 28 02:38:19 dist-test-slave-fz47 krb5kdc[20780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:19.914958 20793 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:38:19.915395 20793 webserver.cc:460] Webserver started at http://127.0.0.1:36667/ using document root <none> and password file <none>
I20260928 02:38:19.915589 20793 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:19.915635 20793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:19.915755 20793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:19.916760 20793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "1c5aa3ae129441eabbe9d433218b024a"
format_stamp: "Formatted at 2026-09-28 02:38:19 on dist-test-slave-fz47"
server_key: "7a2426e0214cc9a830ebd26965c6184d"
server_key_iv: "545e192110fc6b50a9530ba36053c009"
server_key_version: "encryptionkey@0"
I20260928 02:38:19.917097 20793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "1c5aa3ae129441eabbe9d433218b024a"
format_stamp: "Formatted at 2026-09-28 02:38:19 on dist-test-slave-fz47"
server_key: "7a2426e0214cc9a830ebd26965c6184d"
server_key_iv: "545e192110fc6b50a9530ba36053c009"
server_key_version: "encryptionkey@0"
I20260928 02:38:19.918390 20793 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:38:19.919107 20808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:19.919255 20793 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:38:19.919345 20793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "1c5aa3ae129441eabbe9d433218b024a"
format_stamp: "Formatted at 2026-09-28 02:38:19 on dist-test-slave-fz47"
server_key: "7a2426e0214cc9a830ebd26965c6184d"
server_key_iv: "545e192110fc6b50a9530ba36053c009"
server_key_version: "encryptionkey@0"
I20260928 02:38:19.919411 20793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:19.922676 20793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:19.924326 20793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:19.924448 20793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:19.928013 20793 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35879
I20260928 02:38:19.928035 20860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35879 every 8 connection(s)
I20260928 02:38:19.928318 20793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:38:19.929193 20861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:19.930238 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 20793
I20260928 02:38:19.930358 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:38:19.930537 392 external_mini_cluster.cc:1428] Setting key 500e0cca0b66e3821ac1f8434fec3267
I20260928 02:38:19.931691 20861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c5aa3ae129441eabbe9d433218b024a: Bootstrap starting.
I20260928 02:38:19.932291 20861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c5aa3ae129441eabbe9d433218b024a: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:19.932606 20861 log.cc:824] T 00000000000000000000000000000000 P 1c5aa3ae129441eabbe9d433218b024a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:19.933035 20864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:19.931035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60637 (local address 127.0.0.1:35879)
0928 02:38:19.931189 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:19.931196 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:19.931221 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:38:19.931695 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:19.931702 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:19.931728 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:19.931874 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:19.932585 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:19.932589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:19.932590 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:19.932594 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:19.932636 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:19.932646 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:19.932708 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:19.932809 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 02:38:19.932849 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260928 02:38:19.933343 20861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c5aa3ae129441eabbe9d433218b024a: No bootstrap required, opened a new log
I20260928 02:38:19.934032 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:38:19.935613 20861 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c5aa3ae129441eabbe9d433218b024a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c5aa3ae129441eabbe9d433218b024a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35879 } }
I20260928 02:38:19.935762 20861 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c5aa3ae129441eabbe9d433218b024a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:19.935796 20861 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c5aa3ae129441eabbe9d433218b024a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c5aa3ae129441eabbe9d433218b024a, State: Initialized, Role: FOLLOWER
I20260928 02:38:19.935910 20861 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c5aa3ae129441eabbe9d433218b024a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c5aa3ae129441eabbe9d433218b024a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35879 } }
I20260928 02:38:19.936002 20861 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c5aa3ae129441eabbe9d433218b024a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:38:19.936054 20861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c5aa3ae129441eabbe9d433218b024a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:38:19.936100 20861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c5aa3ae129441eabbe9d433218b024a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:19.936278 20864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:19.935296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54447 (local address 127.0.0.1:35879)
0928 02:38:19.935328 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:19.935329 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:19.935449 (+ 120us) server_negotiation.cc:407] Connection header received
0928 02:38:19.935508 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:19.935509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:19.935513 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:19.935563 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:19.936098 (+ 535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:19.936099 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:19.936099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:19.936103 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:19.936116 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:19.936120 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:19.936162 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:19.936210 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 02:38:19.936221 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 02:38:19.936729 20861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c5aa3ae129441eabbe9d433218b024a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c5aa3ae129441eabbe9d433218b024a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35879 } }
I20260928 02:38:19.936827 20861 leader_election.cc:302] T 00000000000000000000000000000000 P 1c5aa3ae129441eabbe9d433218b024a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c5aa3ae129441eabbe9d433218b024a; no voters:
I20260928 02:38:19.936990 20861 leader_election.cc:288] T 00000000000000000000000000000000 P 1c5aa3ae129441eabbe9d433218b024a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:38:19.937023 20871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c5aa3ae129441eabbe9d433218b024a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:19.937176 20871 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c5aa3ae129441eabbe9d433218b024a [term 1 LEADER]: Becoming Leader. State: Replica: 1c5aa3ae129441eabbe9d433218b024a, State: Running, Role: LEADER
I20260928 02:38:19.937225 20861 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c5aa3ae129441eabbe9d433218b024a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:38:19.937330 20871 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c5aa3ae129441eabbe9d433218b024a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c5aa3ae129441eabbe9d433218b024a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35879 } }
I20260928 02:38:19.937667 20872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c5aa3ae129441eabbe9d433218b024a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c5aa3ae129441eabbe9d433218b024a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c5aa3ae129441eabbe9d433218b024a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35879 } } }
I20260928 02:38:19.937667 20873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c5aa3ae129441eabbe9d433218b024a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c5aa3ae129441eabbe9d433218b024a. Latest consensus state: current_term: 1 leader_uuid: "1c5aa3ae129441eabbe9d433218b024a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c5aa3ae129441eabbe9d433218b024a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35879 } } }
I20260928 02:38:19.937826 20872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c5aa3ae129441eabbe9d433218b024a [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:19.937879 20873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c5aa3ae129441eabbe9d433218b024a [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:19.938063 20880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:38:19.938511 20880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:38:19.939951 20880 catalog_manager.cc:1347] Generated new cluster ID: 1a708e3062f242f988054001b2302363
I20260928 02:38:19.939989 20880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:38:19.945919 20880 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:38:19.946051 20880 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:38:19.952170 20880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c5aa3ae129441eabbe9d433218b024a: Generated new TSK 0
I20260928 02:38:19.952329 20880 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:38:19.957849 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 20793
2026-09-28T02:38:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1178 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:38:20 dist-test-slave-fz47 krb5kdc[20892](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:38:20 dist-test-slave-fz47 krb5kdc[20892](info): set up 2 sockets
Sep 28 02:38:20 dist-test-slave-fz47 krb5kdc[20892](info): commencing operation
krb5kdc: starting...
W20260928 02:38:20.904940 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:38:20 dist-test-slave-fz47 krb5kdc[20892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:38:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:38:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:38:21.021976 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46573
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:21.098495 20905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:21.098692 20905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:21.098714 20905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:21.099965 20905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:21.100015 20905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:21.100029 20905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:21.100054 20905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:21.100075 20905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:21.101548 20905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:21.101760 20905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:21.101963 20905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:21.106431 20913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:21.106429 20910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:21.106498 20911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:21.106657 20905 server_base.cc:1034] running on GCE node
I20260928 02:38:21.106951 20905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:21.107225 20905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:21.108359 20905 hybrid_clock.cc:648] HybridClock initialized: now 1790563101108346 us; error 30 us; skew 500 ppm
Sep 28 02:38:21 dist-test-slave-fz47 krb5kdc[20892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:21.110678 20905 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:38:21.111109 20905 webserver.cc:460] Webserver started at http://127.0.0.1:40449/ using document root <none> and password file <none>
I20260928 02:38:21.111297 20905 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:21.111342 20905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:21.111462 20905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:21.112457 20905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "f64f826c985a4612a60d870aada583c8"
format_stamp: "Formatted at 2026-09-28 02:38:21 on dist-test-slave-fz47"
server_key: "d4326673e44ff08f36ccaba9d5631675"
server_key_iv: "6c70b6aade50f3ee41baced078454971"
server_key_version: "encryptionkey@0"
I20260928 02:38:21.112777 20905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "f64f826c985a4612a60d870aada583c8"
format_stamp: "Formatted at 2026-09-28 02:38:21 on dist-test-slave-fz47"
server_key: "d4326673e44ff08f36ccaba9d5631675"
server_key_iv: "6c70b6aade50f3ee41baced078454971"
server_key_version: "encryptionkey@0"
I20260928 02:38:21.114017 20905 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:38:21.114663 20920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:21.114805 20905 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:21.114884 20905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "f64f826c985a4612a60d870aada583c8"
format_stamp: "Formatted at 2026-09-28 02:38:21 on dist-test-slave-fz47"
server_key: "d4326673e44ff08f36ccaba9d5631675"
server_key_iv: "6c70b6aade50f3ee41baced078454971"
server_key_version: "encryptionkey@0"
I20260928 02:38:21.114940 20905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:21.140671 20905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:21.142551 20905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:21.142683 20905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:21.146382 20905 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46573
I20260928 02:38:21.146423 20972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46573 every 8 connection(s)
I20260928 02:38:21.146739 20905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:38:21.147542 20973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:21.149744 20973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f64f826c985a4612a60d870aada583c8: Bootstrap starting.
I20260928 02:38:21.150421 20973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f64f826c985a4612a60d870aada583c8: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:21.150693 20973 log.cc:824] T 00000000000000000000000000000000 P f64f826c985a4612a60d870aada583c8: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:21.151355 20973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f64f826c985a4612a60d870aada583c8: No bootstrap required, opened a new log
I20260928 02:38:21.152921 20973 raft_consensus.cc:348] T 00000000000000000000000000000000 P f64f826c985a4612a60d870aada583c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f64f826c985a4612a60d870aada583c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46573 } }
I20260928 02:38:21.153076 20973 raft_consensus.cc:374] T 00000000000000000000000000000000 P f64f826c985a4612a60d870aada583c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:21.153100 20973 raft_consensus.cc:729] T 00000000000000000000000000000000 P f64f826c985a4612a60d870aada583c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f64f826c985a4612a60d870aada583c8, State: Initialized, Role: FOLLOWER
I20260928 02:38:21.153234 20973 consensus_queue.cc:260] T 00000000000000000000000000000000 P f64f826c985a4612a60d870aada583c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f64f826c985a4612a60d870aada583c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46573 } }
I20260928 02:38:21.153331 20973 raft_consensus.cc:388] T 00000000000000000000000000000000 P f64f826c985a4612a60d870aada583c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:38:21.153358 20973 raft_consensus.cc:482] T 00000000000000000000000000000000 P f64f826c985a4612a60d870aada583c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:38:21.153405 20973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f64f826c985a4612a60d870aada583c8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:21.153888 20973 raft_consensus.cc:504] T 00000000000000000000000000000000 P f64f826c985a4612a60d870aada583c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f64f826c985a4612a60d870aada583c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46573 } }
I20260928 02:38:21.153980 20973 leader_election.cc:302] T 00000000000000000000000000000000 P f64f826c985a4612a60d870aada583c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f64f826c985a4612a60d870aada583c8; no voters:
I20260928 02:38:21.154130 20973 leader_election.cc:288] T 00000000000000000000000000000000 P f64f826c985a4612a60d870aada583c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:38:21.154170 20976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f64f826c985a4612a60d870aada583c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:21.154299 20976 raft_consensus.cc:686] T 00000000000000000000000000000000 P f64f826c985a4612a60d870aada583c8 [term 1 LEADER]: Becoming Leader. State: Replica: f64f826c985a4612a60d870aada583c8, State: Running, Role: LEADER
I20260928 02:38:21.154369 20973 sys_catalog.cc:564] T 00000000000000000000000000000000 P f64f826c985a4612a60d870aada583c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:38:21.154384 20976 consensus_queue.cc:237] T 00000000000000000000000000000000 P f64f826c985a4612a60d870aada583c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f64f826c985a4612a60d870aada583c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46573 } }
I20260928 02:38:21.154760 20978 sys_catalog.cc:455] T 00000000000000000000000000000000 P f64f826c985a4612a60d870aada583c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f64f826c985a4612a60d870aada583c8. Latest consensus state: current_term: 1 leader_uuid: "f64f826c985a4612a60d870aada583c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f64f826c985a4612a60d870aada583c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46573 } } }
I20260928 02:38:21.154882 20977 sys_catalog.cc:455] T 00000000000000000000000000000000 P f64f826c985a4612a60d870aada583c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f64f826c985a4612a60d870aada583c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f64f826c985a4612a60d870aada583c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46573 } } }
I20260928 02:38:21.154978 20977 sys_catalog.cc:458] T 00000000000000000000000000000000 P f64f826c985a4612a60d870aada583c8 [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:21.154891 20978 sys_catalog.cc:458] T 00000000000000000000000000000000 P f64f826c985a4612a60d870aada583c8 [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:21.155388 20985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:38:21.155689 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 20905
I20260928 02:38:21.155804 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:38:21.155918 20985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:38:21.155954 392 external_mini_cluster.cc:1428] Setting key fe184c59ce65daa51ce68183ff493c5f
I20260928 02:38:21.157701 20985 catalog_manager.cc:1347] Generated new cluster ID: c8b01dc475604a4eb79a2019d7a2ce64
I20260928 02:38:21.157752 20985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:38:21 dist-test-slave-fz47 krb5kdc[20892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:38:21.166921 20995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:21.156385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53579 (local address 127.0.0.1:46573)
0928 02:38:21.156576 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:21.156585 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:21.156753 (+ 168us) server_negotiation.cc:407] Connection header received
0928 02:38:21.156871 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:21.156893 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:21.156953 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:21.157078 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:21.157836 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:21.159201 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:21.160230 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:21.160851 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:21.160974 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:21.163614 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:21.163618 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:21.163618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:21.163629 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:21.165421 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:21.165800 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:21.165803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:21.165803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:21.165898 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:21.166084 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:21.166084 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:21.166085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:21.166357 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:21.166467 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:21.166588 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 02:38:21.166708 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":76,"threads_started":1}
I20260928 02:38:21.167852 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:38:21.175783 20995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:21.169159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37883 (local address 127.0.0.1:46573)
0928 02:38:21.169188 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:21.169189 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:21.169315 (+ 126us) server_negotiation.cc:407] Connection header received
0928 02:38:21.169409 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:21.169410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:21.169429 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:21.169481 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:21.170082 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:21.170784 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:21.171776 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:21.172072 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:21.172119 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:21.173364 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:21.173365 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:21.173365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:21.173371 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:21.174749 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:21.175129 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:21.175130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:21.175130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:21.175181 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:21.175349 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:21.175350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:21.175350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:21.175451 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:21.175524 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:21.175637 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 02:38:21.175703 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260928 02:38:21.187183 20985 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:38:21.187919 20985 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:38:21.196774 20985 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f64f826c985a4612a60d870aada583c8: Generated new TSK 0
I20260928 02:38:21.197005 20985 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:38:21.203459 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 20905
2026-09-28T02:38:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1245 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:38:21 dist-test-slave-fz47 krb5kdc[21004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:38:21 dist-test-slave-fz47 krb5kdc[21004](info): set up 2 sockets
Sep 28 02:38:21 dist-test-slave-fz47 krb5kdc[21004](info): commencing operation
krb5kdc: starting...
W20260928 02:38:22.144868 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:38:22 dist-test-slave-fz47 krb5kdc[21004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:38:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:38:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:38:22.250204 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34555
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:22.327924 21017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:22.328120 21017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:22.328141 21017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:22.329279 21017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:22.329319 21017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:22.329330 21017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:22.329341 21017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:22.329351 21017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:22.330803 21017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:22.331024 21017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:22.331218 21017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:22.335599 21023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:22.335736 21022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:22.335731 21025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:22.335713 21017 server_base.cc:1034] running on GCE node
I20260928 02:38:22.336021 21017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:22.336273 21017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:22.337406 21017 hybrid_clock.cc:648] HybridClock initialized: now 1790563102337391 us; error 26 us; skew 500 ppm
Sep 28 02:38:22 dist-test-slave-fz47 krb5kdc[21004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:22.339664 21017 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:38:22.340082 21017 webserver.cc:460] Webserver started at http://127.0.0.1:43167/ using document root <none> and password file <none>
I20260928 02:38:22.340278 21017 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:22.340322 21017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:22.340447 21017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:22.341506 21017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "70ab56e6a282403292bad92949b154cc"
format_stamp: "Formatted at 2026-09-28 02:38:22 on dist-test-slave-fz47"
server_key: "88134e95005e0830d7914d75474d9609"
server_key_iv: "5501208942723947f32aad41fb0c40a3"
server_key_version: "encryptionkey@0"
I20260928 02:38:22.341821 21017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "70ab56e6a282403292bad92949b154cc"
format_stamp: "Formatted at 2026-09-28 02:38:22 on dist-test-slave-fz47"
server_key: "88134e95005e0830d7914d75474d9609"
server_key_iv: "5501208942723947f32aad41fb0c40a3"
server_key_version: "encryptionkey@0"
I20260928 02:38:22.343005 21017 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:38:22.343636 21032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:22.343778 21017 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:22.343856 21017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "70ab56e6a282403292bad92949b154cc"
format_stamp: "Formatted at 2026-09-28 02:38:22 on dist-test-slave-fz47"
server_key: "88134e95005e0830d7914d75474d9609"
server_key_iv: "5501208942723947f32aad41fb0c40a3"
server_key_version: "encryptionkey@0"
I20260928 02:38:22.343915 21017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:22.359532 21017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:22.361317 21017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:22.361455 21017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:22.365172 21017 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34555
I20260928 02:38:22.365212 21084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34555 every 8 connection(s)
I20260928 02:38:22.365552 21017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:38:22.366330 21085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:22.368518 21085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70ab56e6a282403292bad92949b154cc: Bootstrap starting.
I20260928 02:38:22.369043 21085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70ab56e6a282403292bad92949b154cc: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:22.369325 21085 log.cc:824] T 00000000000000000000000000000000 P 70ab56e6a282403292bad92949b154cc: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:22.369935 21085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70ab56e6a282403292bad92949b154cc: No bootstrap required, opened a new log
I20260928 02:38:22.371459 21085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70ab56e6a282403292bad92949b154cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ab56e6a282403292bad92949b154cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34555 } }
I20260928 02:38:22.371594 21085 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70ab56e6a282403292bad92949b154cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:22.371624 21085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70ab56e6a282403292bad92949b154cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70ab56e6a282403292bad92949b154cc, State: Initialized, Role: FOLLOWER
I20260928 02:38:22.371762 21085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70ab56e6a282403292bad92949b154cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ab56e6a282403292bad92949b154cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34555 } }
I20260928 02:38:22.371833 21085 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70ab56e6a282403292bad92949b154cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:38:22.371866 21085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70ab56e6a282403292bad92949b154cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:38:22.371910 21085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70ab56e6a282403292bad92949b154cc [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:22.372390 21085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70ab56e6a282403292bad92949b154cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ab56e6a282403292bad92949b154cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34555 } }
I20260928 02:38:22.372473 21085 leader_election.cc:302] T 00000000000000000000000000000000 P 70ab56e6a282403292bad92949b154cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70ab56e6a282403292bad92949b154cc; no voters:
I20260928 02:38:22.372614 21085 leader_election.cc:288] T 00000000000000000000000000000000 P 70ab56e6a282403292bad92949b154cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:38:22.372665 21088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70ab56e6a282403292bad92949b154cc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:22.372833 21088 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70ab56e6a282403292bad92949b154cc [term 1 LEADER]: Becoming Leader. State: Replica: 70ab56e6a282403292bad92949b154cc, State: Running, Role: LEADER
I20260928 02:38:22.372841 21085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70ab56e6a282403292bad92949b154cc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:38:22.372958 21088 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70ab56e6a282403292bad92949b154cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ab56e6a282403292bad92949b154cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34555 } }
I20260928 02:38:22.373294 21089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70ab56e6a282403292bad92949b154cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70ab56e6a282403292bad92949b154cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ab56e6a282403292bad92949b154cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34555 } } }
I20260928 02:38:22.373389 21089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70ab56e6a282403292bad92949b154cc [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:22.373294 21090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70ab56e6a282403292bad92949b154cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70ab56e6a282403292bad92949b154cc. Latest consensus state: current_term: 1 leader_uuid: "70ab56e6a282403292bad92949b154cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ab56e6a282403292bad92949b154cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34555 } } }
I20260928 02:38:22.373447 21090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70ab56e6a282403292bad92949b154cc [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:22.373596 21092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:38:22.374089 21092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:38:22.374285 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 21017
I20260928 02:38:22.374399 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:38:22.374562 392 external_mini_cluster.cc:1428] Setting key a23964bf2a74221afdbb675f6d67bc23
I20260928 02:38:22.375659 21092 catalog_manager.cc:1347] Generated new cluster ID: 6ee459c888404f7c93bb3abead6453ed
I20260928 02:38:22.375716 21092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:38:22.380993 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:22.375154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59111 (local address 127.0.0.1:34555)
0928 02:38:22.375338 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:22.375348 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:22.375835 (+ 487us) server_negotiation.cc:407] Connection header received
0928 02:38:22.375984 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:22.375991 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:22.376044 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:22.376176 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:22.376412 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:22.377836 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:22.378921 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:22.379531 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:22.379616 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:22.380340 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:22.380344 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:22.380344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:22.380536 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:22.380574 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:22.380588 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:22.380645 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:22.380722 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 02:38:22.380783 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":137,"thread_start_us":130,"threads_started":1}
I20260928 02:38:22.382026 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:38:22.385975 21092 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:38:22.386464 21092 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:38:22.387157 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:22.383518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42543 (local address 127.0.0.1:34555)
0928 02:38:22.383670 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:22.383671 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:22.383764 (+ 93us) server_negotiation.cc:407] Connection header received
0928 02:38:22.383847 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:22.383848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:22.383870 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:22.383897 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:22.384088 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:22.384811 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:22.385871 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:22.386176 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:22.386216 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:22.386918 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:22.386919 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:22.386919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:22.386924 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:22.386940 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:22.386945 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:22.386994 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:22.387049 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 02:38:22.387087 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260928 02:38:22.388581 21092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70ab56e6a282403292bad92949b154cc: Generated new TSK 0
I20260928 02:38:22.388743 21092 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:38:22.393551 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 21017
2026-09-28T02:38:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1190 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:38:22 dist-test-slave-fz47 krb5kdc[21116](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:38:22 dist-test-slave-fz47 krb5kdc[21116](info): set up 2 sockets
Sep 28 02:38:22 dist-test-slave-fz47 krb5kdc[21116](info): commencing operation
krb5kdc: starting...
W20260928 02:38:23.332948 392 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:38:23 dist-test-slave-fz47 krb5kdc[21116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:38:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:38:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:38:23.443645 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35253
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:23.520776 21129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:23.520968 21129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:23.520989 21129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:23.522117 21129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:23.522154 21129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:23.522167 21129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:23.522181 21129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:23.522203 21129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:23.523618 21129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:23.523833 21129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:23.524030 21129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:23.528476 21137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:23.528609 21129 server_base.cc:1034] running on GCE node
W20260928 02:38:23.528478 21134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:23.528568 21135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:23.528875 21129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:23.529126 21129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:23.530275 21129 hybrid_clock.cc:648] HybridClock initialized: now 1790563103530258 us; error 24 us; skew 500 ppm
Sep 28 02:38:23 dist-test-slave-fz47 krb5kdc[21116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:23.532600 21129 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:38:23.532998 21129 webserver.cc:460] Webserver started at http://127.0.0.1:39277/ using document root <none> and password file <none>
I20260928 02:38:23.533202 21129 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:23.533244 21129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:23.533380 21129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:23.534358 21129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "21bbf8dcb1a54b759dd65ff8422fae4a"
format_stamp: "Formatted at 2026-09-28 02:38:23 on dist-test-slave-fz47"
server_key: "eb19f431ccea6e0a038b28096c309b67"
server_key_iv: "f7a6dd780ab47b69aaa723b69ee0b209"
server_key_version: "encryptionkey@0"
I20260928 02:38:23.534660 21129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "21bbf8dcb1a54b759dd65ff8422fae4a"
format_stamp: "Formatted at 2026-09-28 02:38:23 on dist-test-slave-fz47"
server_key: "eb19f431ccea6e0a038b28096c309b67"
server_key_iv: "f7a6dd780ab47b69aaa723b69ee0b209"
server_key_version: "encryptionkey@0"
I20260928 02:38:23.535828 21129 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 02:38:23.536461 21144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:23.536597 21129 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:38:23.536669 21129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "21bbf8dcb1a54b759dd65ff8422fae4a"
format_stamp: "Formatted at 2026-09-28 02:38:23 on dist-test-slave-fz47"
server_key: "eb19f431ccea6e0a038b28096c309b67"
server_key_iv: "f7a6dd780ab47b69aaa723b69ee0b209"
server_key_version: "encryptionkey@0"
I20260928 02:38:23.536743 21129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:23.539793 21129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:23.541416 21129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:23.541524 21129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:23.544934 21129 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35253
I20260928 02:38:23.544966 21196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35253 every 8 connection(s)
I20260928 02:38:23.545224 21129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:38:23.546104 21197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:23.547369 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 21129
I20260928 02:38:23.547464 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:38:23.547622 392 external_mini_cluster.cc:1428] Setting key c133de1be6c0442029a10223461ab14d
I20260928 02:38:23.548395 21197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21bbf8dcb1a54b759dd65ff8422fae4a: Bootstrap starting.
I20260928 02:38:23.548905 21197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21bbf8dcb1a54b759dd65ff8422fae4a: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:23.549154 21197 log.cc:824] T 00000000000000000000000000000000 P 21bbf8dcb1a54b759dd65ff8422fae4a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:23.549791 21200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:23.548051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60449 (local address 127.0.0.1:35253)
0928 02:38:23.548231 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:23.548239 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:23.548263 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:38:23.548664 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:23.548671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:23.548697 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:23.548818 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:23.549456 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:23.549459 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:23.549459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:23.549462 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:23.549492 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:23.549502 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:23.549571 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:23.549621 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 02:38:23.549659 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
I20260928 02:38:23.549841 21197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21bbf8dcb1a54b759dd65ff8422fae4a: No bootstrap required, opened a new log
I20260928 02:38:23.550609 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:38:23.551862 21197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21bbf8dcb1a54b759dd65ff8422fae4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21bbf8dcb1a54b759dd65ff8422fae4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35253 } }
I20260928 02:38:23.552012 21197 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21bbf8dcb1a54b759dd65ff8422fae4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:23.552044 21197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21bbf8dcb1a54b759dd65ff8422fae4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21bbf8dcb1a54b759dd65ff8422fae4a, State: Initialized, Role: FOLLOWER
I20260928 02:38:23.552156 21197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21bbf8dcb1a54b759dd65ff8422fae4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21bbf8dcb1a54b759dd65ff8422fae4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35253 } }
I20260928 02:38:23.552232 21197 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21bbf8dcb1a54b759dd65ff8422fae4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:38:23.552269 21197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21bbf8dcb1a54b759dd65ff8422fae4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:38:23.552314 21197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21bbf8dcb1a54b759dd65ff8422fae4a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:23.552795 21197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21bbf8dcb1a54b759dd65ff8422fae4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21bbf8dcb1a54b759dd65ff8422fae4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35253 } }
I20260928 02:38:23.552886 21197 leader_election.cc:302] T 00000000000000000000000000000000 P 21bbf8dcb1a54b759dd65ff8422fae4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21bbf8dcb1a54b759dd65ff8422fae4a; no voters:
I20260928 02:38:23.553036 21197 leader_election.cc:288] T 00000000000000000000000000000000 P 21bbf8dcb1a54b759dd65ff8422fae4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:38:23.553074 21207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21bbf8dcb1a54b759dd65ff8422fae4a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:23.553206 21207 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21bbf8dcb1a54b759dd65ff8422fae4a [term 1 LEADER]: Becoming Leader. State: Replica: 21bbf8dcb1a54b759dd65ff8422fae4a, State: Running, Role: LEADER
I20260928 02:38:23.553278 21197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21bbf8dcb1a54b759dd65ff8422fae4a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:38:23.553306 21207 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21bbf8dcb1a54b759dd65ff8422fae4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21bbf8dcb1a54b759dd65ff8422fae4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35253 } }
I20260928 02:38:23.553629 21208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21bbf8dcb1a54b759dd65ff8422fae4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21bbf8dcb1a54b759dd65ff8422fae4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21bbf8dcb1a54b759dd65ff8422fae4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35253 } } }
I20260928 02:38:23.553651 21209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21bbf8dcb1a54b759dd65ff8422fae4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21bbf8dcb1a54b759dd65ff8422fae4a. Latest consensus state: current_term: 1 leader_uuid: "21bbf8dcb1a54b759dd65ff8422fae4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21bbf8dcb1a54b759dd65ff8422fae4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35253 } } }
I20260928 02:38:23.553719 21208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21bbf8dcb1a54b759dd65ff8422fae4a [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:23.553746 21209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21bbf8dcb1a54b759dd65ff8422fae4a [sys.catalog]: This master's current role is: LEADER
W20260928 02:38:23.554929 21223 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 21bbf8dcb1a54b759dd65ff8422fae4a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:38:23.555248 21223 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:38:23.555332 21215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:38:23.555513 21215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:38:23.555598 21200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:23.551831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34767 (local address 127.0.0.1:35253)
0928 02:38:23.551863 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:23.551865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:23.554688 (+ 2823us) server_negotiation.cc:407] Connection header received
0928 02:38:23.554761 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:23.554761 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:23.554766 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:23.554809 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:23.555374 (+ 565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:23.555375 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:23.555375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:23.555380 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:23.555394 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:23.555398 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:23.555441 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:23.555525 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 02:38:23.555537 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 02:38:23.556839 21215 catalog_manager.cc:1347] Generated new cluster ID: b63167fb08e44667b9d1f53c2b1d90fd
I20260928 02:38:23.556888 21215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:38:23.565591 21215 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:38:23.565728 21215 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:38:23.570340 21215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21bbf8dcb1a54b759dd65ff8422fae4a: Generated new TSK 0
I20260928 02:38:23.570482 21215 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:38:23.573796 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 21129
2026-09-28T02:38:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1180 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:38:23 dist-test-slave-fz47 krb5kdc[21228](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:38:23 dist-test-slave-fz47 krb5kdc[21228](info): set up 2 sockets
Sep 28 02:38:23 dist-test-slave-fz47 krb5kdc[21228](info): commencing operation
krb5kdc: starting...
W20260928 02:38:24.508920 392 mini_kdc.cc:121] Time spent starting KDC: real 0.918s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:38:24 dist-test-slave-fz47 krb5kdc[21228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:38:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:38:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:38:24.615636 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32997
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:24.691797 21241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:24.691980 21241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:24.692008 21241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:24.693105 21241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:24.693142 21241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:24.693161 21241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:24.693176 21241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:24.693192 21241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:24.694622 21241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:24.694845 21241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:24.695024 21241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:24.699360 21246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:24.699462 21241 server_base.cc:1034] running on GCE node
W20260928 02:38:24.699455 21249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:24.699385 21247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:24.699800 21241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:24.700023 21241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:24.701150 21241 hybrid_clock.cc:648] HybridClock initialized: now 1790563104701136 us; error 28 us; skew 500 ppm
Sep 28 02:38:24 dist-test-slave-fz47 krb5kdc[21228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:24.708199 21241 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:38:24.708595 21241 webserver.cc:460] Webserver started at http://127.0.0.1:42785/ using document root <none> and password file <none>
I20260928 02:38:24.708781 21241 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:24.708820 21241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:24.708926 21241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:24.709946 21241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "d70419f85d3d4dd8ab3bacbb8c626541"
format_stamp: "Formatted at 2026-09-28 02:38:24 on dist-test-slave-fz47"
server_key: "910b3ae2716061821ffe00a95f349172"
server_key_iv: "1568fe0f0c03b7adc2c8c58b6bcfa968"
server_key_version: "encryptionkey@0"
I20260928 02:38:24.710246 21241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "d70419f85d3d4dd8ab3bacbb8c626541"
format_stamp: "Formatted at 2026-09-28 02:38:24 on dist-test-slave-fz47"
server_key: "910b3ae2716061821ffe00a95f349172"
server_key_iv: "1568fe0f0c03b7adc2c8c58b6bcfa968"
server_key_version: "encryptionkey@0"
I20260928 02:38:24.711380 21241 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:38:24.711980 21256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:24.712117 21241 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:24.712186 21241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "d70419f85d3d4dd8ab3bacbb8c626541"
format_stamp: "Formatted at 2026-09-28 02:38:24 on dist-test-slave-fz47"
server_key: "910b3ae2716061821ffe00a95f349172"
server_key_iv: "1568fe0f0c03b7adc2c8c58b6bcfa968"
server_key_version: "encryptionkey@0"
I20260928 02:38:24.712249 21241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:24.733034 21241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:24.734724 21241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:24.734849 21241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:24.738471 21241 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32997
I20260928 02:38:24.738508 21308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32997 every 8 connection(s)
I20260928 02:38:24.738837 21241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:38:24.739368 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 21241
I20260928 02:38:24.739467 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:38:24.739634 392 external_mini_cluster.cc:1428] Setting key bb2110c85b4a4ba835d42a83751ebb58
I20260928 02:38:24.739631 21309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:24.742614 21309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d70419f85d3d4dd8ab3bacbb8c626541: Bootstrap starting.
I20260928 02:38:24.743274 21309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d70419f85d3d4dd8ab3bacbb8c626541: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:24.743554 21309 log.cc:824] T 00000000000000000000000000000000 P d70419f85d3d4dd8ab3bacbb8c626541: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:24.744166 21309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d70419f85d3d4dd8ab3bacbb8c626541: No bootstrap required, opened a new log
Sep 28 02:38:24 dist-test-slave-fz47 krb5kdc[21228](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:38:24.746349 21309 raft_consensus.cc:348] T 00000000000000000000000000000000 P d70419f85d3d4dd8ab3bacbb8c626541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70419f85d3d4dd8ab3bacbb8c626541" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } }
I20260928 02:38:24.746505 21309 raft_consensus.cc:374] T 00000000000000000000000000000000 P d70419f85d3d4dd8ab3bacbb8c626541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:24.746541 21309 raft_consensus.cc:729] T 00000000000000000000000000000000 P d70419f85d3d4dd8ab3bacbb8c626541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d70419f85d3d4dd8ab3bacbb8c626541, State: Initialized, Role: FOLLOWER
I20260928 02:38:24.746665 21309 consensus_queue.cc:260] T 00000000000000000000000000000000 P d70419f85d3d4dd8ab3bacbb8c626541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70419f85d3d4dd8ab3bacbb8c626541" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } }
I20260928 02:38:24.746726 21309 raft_consensus.cc:388] T 00000000000000000000000000000000 P d70419f85d3d4dd8ab3bacbb8c626541 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:38:24.746754 21309 raft_consensus.cc:482] T 00000000000000000000000000000000 P d70419f85d3d4dd8ab3bacbb8c626541 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:38:24.746806 21309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d70419f85d3d4dd8ab3bacbb8c626541 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:24.747264 21309 raft_consensus.cc:504] T 00000000000000000000000000000000 P d70419f85d3d4dd8ab3bacbb8c626541 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70419f85d3d4dd8ab3bacbb8c626541" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } }
I20260928 02:38:24.747355 21309 leader_election.cc:302] T 00000000000000000000000000000000 P d70419f85d3d4dd8ab3bacbb8c626541 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d70419f85d3d4dd8ab3bacbb8c626541; no voters:
I20260928 02:38:24.747501 21309 leader_election.cc:288] T 00000000000000000000000000000000 P d70419f85d3d4dd8ab3bacbb8c626541 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:38:24.747540 21314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d70419f85d3d4dd8ab3bacbb8c626541 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:24.747676 21314 raft_consensus.cc:686] T 00000000000000000000000000000000 P d70419f85d3d4dd8ab3bacbb8c626541 [term 1 LEADER]: Becoming Leader. State: Replica: d70419f85d3d4dd8ab3bacbb8c626541, State: Running, Role: LEADER
I20260928 02:38:24.747741 21309 sys_catalog.cc:564] T 00000000000000000000000000000000 P d70419f85d3d4dd8ab3bacbb8c626541 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:38:24.747799 21314 consensus_queue.cc:237] T 00000000000000000000000000000000 P d70419f85d3d4dd8ab3bacbb8c626541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70419f85d3d4dd8ab3bacbb8c626541" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } }
I20260928 02:38:24.748533 21316 sys_catalog.cc:455] T 00000000000000000000000000000000 P d70419f85d3d4dd8ab3bacbb8c626541 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d70419f85d3d4dd8ab3bacbb8c626541" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70419f85d3d4dd8ab3bacbb8c626541" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } } }
I20260928 02:38:24.748656 21316 sys_catalog.cc:458] T 00000000000000000000000000000000 P d70419f85d3d4dd8ab3bacbb8c626541 [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:24.748685 21315 sys_catalog.cc:455] T 00000000000000000000000000000000 P d70419f85d3d4dd8ab3bacbb8c626541 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d70419f85d3d4dd8ab3bacbb8c626541. Latest consensus state: current_term: 1 leader_uuid: "d70419f85d3d4dd8ab3bacbb8c626541" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70419f85d3d4dd8ab3bacbb8c626541" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } } }
I20260928 02:38:24.748745 21315 sys_catalog.cc:458] T 00000000000000000000000000000000 P d70419f85d3d4dd8ab3bacbb8c626541 [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:24.748924 21327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:38:24.749480 21327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:38:24.749778 21312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:24.740098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.64:34813 (local address 127.0.0.1:32997)
0928 02:38:24.740242 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:24.740249 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:24.740275 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:38:24.740644 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:24.740650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:24.740693 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:24.740805 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:24.741540 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:24.742400 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:24.743377 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:24.743784 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:24.743862 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:24.746184 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:24.746188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:24.746189 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:24.746199 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:24.747890 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:24.748292 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:24.748295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:24.748296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:24.748415 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:24.748868 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:24.748870 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:24.748870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:24.749156 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:24.749341 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:24.749472 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 02:38:24.749565 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260928 02:38:24.750617 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:38:24.751067 21327 catalog_manager.cc:1347] Generated new cluster ID: 1c65fdcdce7e4293b16668e8d1e8a515
I20260928 02:38:24.751117 21327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:38:24.755216 21327 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:38:24.755856 21327 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:38:24.758929 21312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:24.751998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.64:39849 (local address 127.0.0.1:32997)
0928 02:38:24.752028 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:24.752029 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:24.752138 (+ 109us) server_negotiation.cc:407] Connection header received
0928 02:38:24.752184 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:24.752185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:24.752194 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:24.752248 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:24.752832 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:24.753546 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:24.754547 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:24.754841 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:24.754909 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:24.756268 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:24.756270 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:24.756270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:24.756276 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:24.757855 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:24.758197 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:24.758198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:24.758199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:24.758255 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:24.758453 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:24.758454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:24.758455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:24.758578 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:24.758681 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:24.758787 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 02:38:24.758847 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 02:38:24.760723 21327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d70419f85d3d4dd8ab3bacbb8c626541: Generated new TSK 0
I20260928 02:38:24.760850 21327 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:38:24.763195 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 21241
W20260928 02:38:24.766716 21226 connection.cc:535] client connection to 127.0.0.1:32997 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32997): Connection reset by peer (error 104)
2026-09-28T02:38:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1188 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:38:24 dist-test-slave-fz47 krb5kdc[21340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:38:24 dist-test-slave-fz47 krb5kdc[21340](info): set up 2 sockets
Sep 28 02:38:24 dist-test-slave-fz47 krb5kdc[21340](info): commencing operation
krb5kdc: starting...
W20260928 02:38:25.700917 392 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:38:25 dist-test-slave-fz47 krb5kdc[21340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:38:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:38:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:38:25.805177 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45931
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:25.880654 21353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:25.880847 21353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:25.880868 21353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:25.881983 21353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:25.882021 21353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:25.882033 21353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:25.882045 21353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:25.882062 21353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:25.883469 21353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:25.883682 21353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:25.883877 21353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:25.888190 21358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:25.888239 21361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:25.888315 21359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:25.888326 21353 server_base.cc:1034] running on GCE node
I20260928 02:38:25.888689 21353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:25.888918 21353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:25.890065 21353 hybrid_clock.cc:648] HybridClock initialized: now 1790563105890052 us; error 28 us; skew 500 ppm
Sep 28 02:38:25 dist-test-slave-fz47 krb5kdc[21340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:25.892321 21353 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:38:25.892686 21353 webserver.cc:460] Webserver started at http://127.0.0.1:46179/ using document root <none> and password file <none>
I20260928 02:38:25.892876 21353 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:25.892920 21353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:25.893034 21353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:25.894017 21353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "bad8d6105d6546519b6b63bf90960a2c"
format_stamp: "Formatted at 2026-09-28 02:38:25 on dist-test-slave-fz47"
server_key: "e2b72d2ebe1f089810c193b26ac90076"
server_key_iv: "d664c64413664d5e5cdad334ba3b87cb"
server_key_version: "encryptionkey@0"
I20260928 02:38:25.894332 21353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "bad8d6105d6546519b6b63bf90960a2c"
format_stamp: "Formatted at 2026-09-28 02:38:25 on dist-test-slave-fz47"
server_key: "e2b72d2ebe1f089810c193b26ac90076"
server_key_iv: "d664c64413664d5e5cdad334ba3b87cb"
server_key_version: "encryptionkey@0"
I20260928 02:38:25.895470 21353 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 02:38:25.896044 21368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:25.896188 21353 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:38:25.896273 21353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "bad8d6105d6546519b6b63bf90960a2c"
format_stamp: "Formatted at 2026-09-28 02:38:25 on dist-test-slave-fz47"
server_key: "e2b72d2ebe1f089810c193b26ac90076"
server_key_iv: "d664c64413664d5e5cdad334ba3b87cb"
server_key_version: "encryptionkey@0"
I20260928 02:38:25.896337 21353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:25.928686 21353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:25.930424 21353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:25.930552 21353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:25.934321 21353 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45931
I20260928 02:38:25.934356 21420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45931 every 8 connection(s)
I20260928 02:38:25.934662 21353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:38:25.935448 21421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:25.937667 21421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bad8d6105d6546519b6b63bf90960a2c: Bootstrap starting.
I20260928 02:38:25.938198 21421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bad8d6105d6546519b6b63bf90960a2c: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:25.938437 21421 log.cc:824] T 00000000000000000000000000000000 P bad8d6105d6546519b6b63bf90960a2c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:25.939036 21421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bad8d6105d6546519b6b63bf90960a2c: No bootstrap required, opened a new log
I20260928 02:38:25.939162 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 21353
I20260928 02:38:25.939263 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:38:25.939422 392 external_mini_cluster.cc:1428] Setting key c89d0704943522b23aebb99840e32a5c
I20260928 02:38:25.940995 21421 raft_consensus.cc:348] T 00000000000000000000000000000000 P bad8d6105d6546519b6b63bf90960a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bad8d6105d6546519b6b63bf90960a2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45931 } }
I20260928 02:38:25.941157 21421 raft_consensus.cc:374] T 00000000000000000000000000000000 P bad8d6105d6546519b6b63bf90960a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:25.941192 21421 raft_consensus.cc:729] T 00000000000000000000000000000000 P bad8d6105d6546519b6b63bf90960a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bad8d6105d6546519b6b63bf90960a2c, State: Initialized, Role: FOLLOWER
I20260928 02:38:25.941375 21421 consensus_queue.cc:260] T 00000000000000000000000000000000 P bad8d6105d6546519b6b63bf90960a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bad8d6105d6546519b6b63bf90960a2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45931 } }
I20260928 02:38:25.941460 21421 raft_consensus.cc:388] T 00000000000000000000000000000000 P bad8d6105d6546519b6b63bf90960a2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:38:25.941500 21421 raft_consensus.cc:482] T 00000000000000000000000000000000 P bad8d6105d6546519b6b63bf90960a2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:38:25.941548 21421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bad8d6105d6546519b6b63bf90960a2c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:25.942185 21421 raft_consensus.cc:504] T 00000000000000000000000000000000 P bad8d6105d6546519b6b63bf90960a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bad8d6105d6546519b6b63bf90960a2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45931 } }
I20260928 02:38:25.942287 21421 leader_election.cc:302] T 00000000000000000000000000000000 P bad8d6105d6546519b6b63bf90960a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bad8d6105d6546519b6b63bf90960a2c; no voters:
I20260928 02:38:25.942425 21421 leader_election.cc:288] T 00000000000000000000000000000000 P bad8d6105d6546519b6b63bf90960a2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:38:25.942469 21426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bad8d6105d6546519b6b63bf90960a2c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:25.942600 21426 raft_consensus.cc:686] T 00000000000000000000000000000000 P bad8d6105d6546519b6b63bf90960a2c [term 1 LEADER]: Becoming Leader. State: Replica: bad8d6105d6546519b6b63bf90960a2c, State: Running, Role: LEADER
I20260928 02:38:25.942642 21421 sys_catalog.cc:564] T 00000000000000000000000000000000 P bad8d6105d6546519b6b63bf90960a2c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:38:25.942745 21426 consensus_queue.cc:237] T 00000000000000000000000000000000 P bad8d6105d6546519b6b63bf90960a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bad8d6105d6546519b6b63bf90960a2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45931 } }
I20260928 02:38:25.943132 21427 sys_catalog.cc:455] T 00000000000000000000000000000000 P bad8d6105d6546519b6b63bf90960a2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bad8d6105d6546519b6b63bf90960a2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bad8d6105d6546519b6b63bf90960a2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45931 } } }
I20260928 02:38:25.943217 21427 sys_catalog.cc:458] T 00000000000000000000000000000000 P bad8d6105d6546519b6b63bf90960a2c [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:25.943360 21428 sys_catalog.cc:455] T 00000000000000000000000000000000 P bad8d6105d6546519b6b63bf90960a2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bad8d6105d6546519b6b63bf90960a2c. Latest consensus state: current_term: 1 leader_uuid: "bad8d6105d6546519b6b63bf90960a2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bad8d6105d6546519b6b63bf90960a2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45931 } } }
I20260928 02:38:25.943427 21428 sys_catalog.cc:458] T 00000000000000000000000000000000 P bad8d6105d6546519b6b63bf90960a2c [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:25.943884 21433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:38:25.944386 21433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:38:25.946120 21425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:25.939862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.64:53601 (local address 127.0.0.1:45931)
0928 02:38:25.940045 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:25.940054 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:25.940087 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:38:25.940963 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:25.940971 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:25.941038 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:25.941179 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:25.941404 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:25.942974 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:25.944105 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:25.944626 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:25.945012 (+ 386us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:25.945520 (+ 508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:25.945523 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:25.945523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:25.945682 (+ 159us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:25.945718 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:25.945728 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:25.945791 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:25.945901 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 02:38:25.945923 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
I20260928 02:38:25.947034 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:38:25.950850 21425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:25.948126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.64:57293 (local address 127.0.0.1:45931)
0928 02:38:25.948158 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:25.948159 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:25.948271 (+ 112us) server_negotiation.cc:407] Connection header received
0928 02:38:25.948324 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:25.948325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:25.948332 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:25.948374 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:25.948481 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:25.948918 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:25.949880 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:25.950074 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:25.950119 (+ 45us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:25.950655 (+ 536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:25.950655 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:25.950656 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:25.950659 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:25.950670 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:25.950673 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:25.950716 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:25.950795 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 02:38:25.950804 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 02:38:25.953670 21433 catalog_manager.cc:1347] Generated new cluster ID: a9455f3cc0d746e8ad86ba8c49b93ed4
I20260928 02:38:25.953712 21433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:38:25.962044 21433 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:38:25.962790 21433 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:38:25.969511 21433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bad8d6105d6546519b6b63bf90960a2c: Generated new TSK 0
I20260928 02:38:25.969655 21433 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:38:25.977432 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 21353
W20260928 02:38:25.980973 21338 connection.cc:535] client connection to 127.0.0.1:45931 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45931): Connection reset by peer (error 104)
2026-09-28T02:38:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1214 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:38:26 dist-test-slave-fz47 krb5kdc[21452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:38:26 dist-test-slave-fz47 krb5kdc[21452](info): set up 2 sockets
Sep 28 02:38:26 dist-test-slave-fz47 krb5kdc[21452](info): commencing operation
krb5kdc: starting...
W20260928 02:38:26.916991 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:38:26 dist-test-slave-fz47 krb5kdc[21452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:38:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:38:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:38:27.023829 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42605
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:27.102270 21465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:27.102454 21465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:27.102474 21465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:27.103577 21465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:27.103615 21465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:27.103627 21465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:27.103638 21465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:27.103652 21465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:27.105069 21465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:27.105293 21465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:27.105489 21465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:38:27.109946 21465 server_base.cc:1034] running on GCE node
W20260928 02:38:27.109910 21470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:27.109978 21473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:27.109918 21471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:27.110344 21465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:27.110600 21465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:27.111735 21465 hybrid_clock.cc:648] HybridClock initialized: now 1790563107111724 us; error 31 us; skew 500 ppm
Sep 28 02:38:27 dist-test-slave-fz47 krb5kdc[21452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:27.114063 21465 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:38:27.114468 21465 webserver.cc:460] Webserver started at http://127.0.0.1:38963/ using document root <none> and password file <none>
I20260928 02:38:27.114668 21465 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:27.114714 21465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:27.114831 21465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:27.115818 21465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "ae7fab8ced14486085b7f0e46f0c418a"
format_stamp: "Formatted at 2026-09-28 02:38:27 on dist-test-slave-fz47"
server_key: "f6c1d99131a4773b7cab3f6fa9051267"
server_key_iv: "851bc9d27a2c5d24a96d9e19285e5348"
server_key_version: "encryptionkey@0"
I20260928 02:38:27.116143 21465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "ae7fab8ced14486085b7f0e46f0c418a"
format_stamp: "Formatted at 2026-09-28 02:38:27 on dist-test-slave-fz47"
server_key: "f6c1d99131a4773b7cab3f6fa9051267"
server_key_iv: "851bc9d27a2c5d24a96d9e19285e5348"
server_key_version: "encryptionkey@0"
I20260928 02:38:27.117363 21465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:38:27.118032 21480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:27.118188 21465 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 02:38:27.118259 21465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "ae7fab8ced14486085b7f0e46f0c418a"
format_stamp: "Formatted at 2026-09-28 02:38:27 on dist-test-slave-fz47"
server_key: "f6c1d99131a4773b7cab3f6fa9051267"
server_key_iv: "851bc9d27a2c5d24a96d9e19285e5348"
server_key_version: "encryptionkey@0"
I20260928 02:38:27.118325 21465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:27.121546 21465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:27.123183 21465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:27.123299 21465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:27.126783 21465 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42605
I20260928 02:38:27.126813 21532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42605 every 8 connection(s)
I20260928 02:38:27.127089 21465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:38:27.127760 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 21465
I20260928 02:38:27.127866 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:38:27.127979 21533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:27.128033 392 external_mini_cluster.cc:1428] Setting key dcebf3bb1b8e5d1156811545832f384d
I20260928 02:38:27.130359 21536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:27.128498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.64:53673 (local address 127.0.0.1:42605)
0928 02:38:27.128655 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:27.128664 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:27.128695 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:38:27.129245 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:27.129275 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:27.129312 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:27.129461 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:27.129954 (+ 493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:27.129957 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:27.129958 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:27.129961 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:27.130002 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:27.130017 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:27.130098 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:27.130141 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 02:38:27.130181 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260928 02:38:27.131142 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:38:27.131175 21533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae7fab8ced14486085b7f0e46f0c418a: Bootstrap starting.
I20260928 02:38:27.131705 21533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae7fab8ced14486085b7f0e46f0c418a: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:27.131963 21533 log.cc:824] T 00000000000000000000000000000000 P ae7fab8ced14486085b7f0e46f0c418a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:27.132673 21533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae7fab8ced14486085b7f0e46f0c418a: No bootstrap required, opened a new log
I20260928 02:38:27.133251 21536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:27.132345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.64:48283 (local address 127.0.0.1:42605)
0928 02:38:27.132377 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:27.132379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:27.132505 (+ 126us) server_negotiation.cc:407] Connection header received
0928 02:38:27.132565 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:27.132566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:27.132570 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:27.132610 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:27.133082 (+ 472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:27.133083 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:27.133083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:27.133086 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:27.133097 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:27.133101 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:27.133138 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:27.133189 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 02:38:27.133199 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 02:38:27.134639 21533 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae7fab8ced14486085b7f0e46f0c418a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae7fab8ced14486085b7f0e46f0c418a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42605 } }
I20260928 02:38:27.134765 21533 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae7fab8ced14486085b7f0e46f0c418a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:27.134785 21533 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae7fab8ced14486085b7f0e46f0c418a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae7fab8ced14486085b7f0e46f0c418a, State: Initialized, Role: FOLLOWER
I20260928 02:38:27.134894 21533 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae7fab8ced14486085b7f0e46f0c418a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae7fab8ced14486085b7f0e46f0c418a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42605 } }
I20260928 02:38:27.134984 21533 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae7fab8ced14486085b7f0e46f0c418a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:38:27.135025 21533 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae7fab8ced14486085b7f0e46f0c418a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:38:27.135069 21533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae7fab8ced14486085b7f0e46f0c418a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:27.135517 21533 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae7fab8ced14486085b7f0e46f0c418a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae7fab8ced14486085b7f0e46f0c418a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42605 } }
I20260928 02:38:27.135599 21533 leader_election.cc:302] T 00000000000000000000000000000000 P ae7fab8ced14486085b7f0e46f0c418a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae7fab8ced14486085b7f0e46f0c418a; no voters:
I20260928 02:38:27.135730 21533 leader_election.cc:288] T 00000000000000000000000000000000 P ae7fab8ced14486085b7f0e46f0c418a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:38:27.135783 21543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae7fab8ced14486085b7f0e46f0c418a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:27.135931 21533 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae7fab8ced14486085b7f0e46f0c418a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:38:27.135939 21543 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae7fab8ced14486085b7f0e46f0c418a [term 1 LEADER]: Becoming Leader. State: Replica: ae7fab8ced14486085b7f0e46f0c418a, State: Running, Role: LEADER
I20260928 02:38:27.136042 21543 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae7fab8ced14486085b7f0e46f0c418a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae7fab8ced14486085b7f0e46f0c418a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42605 } }
I20260928 02:38:27.136365 21544 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae7fab8ced14486085b7f0e46f0c418a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae7fab8ced14486085b7f0e46f0c418a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae7fab8ced14486085b7f0e46f0c418a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42605 } } }
I20260928 02:38:27.136452 21544 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae7fab8ced14486085b7f0e46f0c418a [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:27.136632 21545 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae7fab8ced14486085b7f0e46f0c418a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae7fab8ced14486085b7f0e46f0c418a. Latest consensus state: current_term: 1 leader_uuid: "ae7fab8ced14486085b7f0e46f0c418a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae7fab8ced14486085b7f0e46f0c418a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42605 } } }
I20260928 02:38:27.136694 21551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:38:27.137215 21551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:38:27.136718 21545 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae7fab8ced14486085b7f0e46f0c418a [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:27.138700 21551 catalog_manager.cc:1347] Generated new cluster ID: f4b06cd6393f4368af0de61c57752b16
I20260928 02:38:27.138738 21551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:38:27.147352 21551 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:38:27.147497 21551 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:38:27.155491 21551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae7fab8ced14486085b7f0e46f0c418a: Generated new TSK 0
I20260928 02:38:27.155660 21551 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:38:27.157805 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 21465
2026-09-28T02:38:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1180 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:38:27 dist-test-slave-fz47 krb5kdc[21564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:38:27 dist-test-slave-fz47 krb5kdc[21564](info): set up 2 sockets
Sep 28 02:38:27 dist-test-slave-fz47 krb5kdc[21564](info): commencing operation
krb5kdc: starting...
W20260928 02:38:28.105046 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:38:28 dist-test-slave-fz47 krb5kdc[21564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:38:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:38:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:38:28.211695 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45653
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:28.289108 21577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:28.289314 21577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:28.289345 21577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:28.290486 21577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:28.290527 21577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:28.290545 21577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:28.290561 21577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:28.290576 21577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:28.292001 21577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45653
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:28.292227 21577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:28.292430 21577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:28.297235 21583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:28.297242 21585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:28.297242 21582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:28.297317 21577 server_base.cc:1034] running on GCE node
I20260928 02:38:28.297672 21577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:28.297942 21577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:28.299098 21577 hybrid_clock.cc:648] HybridClock initialized: now 1790563108299084 us; error 29 us; skew 500 ppm
Sep 28 02:38:28 dist-test-slave-fz47 krb5kdc[21564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:28.301414 21577 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:38:28.301869 21577 webserver.cc:460] Webserver started at http://127.0.0.1:42425/ using document root <none> and password file <none>
I20260928 02:38:28.302058 21577 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:28.302102 21577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:28.302222 21577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:28.303180 21577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "18e64090fe264086830c635c6034f3cb"
format_stamp: "Formatted at 2026-09-28 02:38:28 on dist-test-slave-fz47"
server_key: "6c105574a194f85df208b52d67afd0c6"
server_key_iv: "1ab2c803f04a55b8dac7a4083b42f9f1"
server_key_version: "encryptionkey@0"
I20260928 02:38:28.303499 21577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "18e64090fe264086830c635c6034f3cb"
format_stamp: "Formatted at 2026-09-28 02:38:28 on dist-test-slave-fz47"
server_key: "6c105574a194f85df208b52d67afd0c6"
server_key_iv: "1ab2c803f04a55b8dac7a4083b42f9f1"
server_key_version: "encryptionkey@0"
I20260928 02:38:28.304629 21577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:38:28.305217 21592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:28.305370 21577 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:28.305464 21577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "18e64090fe264086830c635c6034f3cb"
format_stamp: "Formatted at 2026-09-28 02:38:28 on dist-test-slave-fz47"
server_key: "6c105574a194f85df208b52d67afd0c6"
server_key_iv: "1ab2c803f04a55b8dac7a4083b42f9f1"
server_key_version: "encryptionkey@0"
I20260928 02:38:28.305531 21577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:28.321868 21577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:28.323643 21577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:28.323760 21577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:28.327365 21577 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45653
I20260928 02:38:28.327396 21644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45653 every 8 connection(s)
I20260928 02:38:28.327685 21577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:38:28.328437 21645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:28.330610 21645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18e64090fe264086830c635c6034f3cb: Bootstrap starting.
I20260928 02:38:28.331099 21645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18e64090fe264086830c635c6034f3cb: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:28.331310 21645 log.cc:824] T 00000000000000000000000000000000 P 18e64090fe264086830c635c6034f3cb: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:28.331885 21645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18e64090fe264086830c635c6034f3cb: No bootstrap required, opened a new log
I20260928 02:38:28.333396 21645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18e64090fe264086830c635c6034f3cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18e64090fe264086830c635c6034f3cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45653 } }
I20260928 02:38:28.333520 21645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18e64090fe264086830c635c6034f3cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:28.333540 21645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18e64090fe264086830c635c6034f3cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18e64090fe264086830c635c6034f3cb, State: Initialized, Role: FOLLOWER
I20260928 02:38:28.333664 21645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18e64090fe264086830c635c6034f3cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18e64090fe264086830c635c6034f3cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45653 } }
I20260928 02:38:28.333736 21645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18e64090fe264086830c635c6034f3cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:38:28.333763 21645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18e64090fe264086830c635c6034f3cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:38:28.333793 21645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18e64090fe264086830c635c6034f3cb [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:28.334220 21645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18e64090fe264086830c635c6034f3cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18e64090fe264086830c635c6034f3cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45653 } }
I20260928 02:38:28.334298 21645 leader_election.cc:302] T 00000000000000000000000000000000 P 18e64090fe264086830c635c6034f3cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18e64090fe264086830c635c6034f3cb; no voters:
I20260928 02:38:28.334406 21645 leader_election.cc:288] T 00000000000000000000000000000000 P 18e64090fe264086830c635c6034f3cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:38:28.334453 21648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18e64090fe264086830c635c6034f3cb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:28.334599 21648 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18e64090fe264086830c635c6034f3cb [term 1 LEADER]: Becoming Leader. State: Replica: 18e64090fe264086830c635c6034f3cb, State: Running, Role: LEADER
I20260928 02:38:28.334640 21645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18e64090fe264086830c635c6034f3cb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:38:28.334689 21648 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18e64090fe264086830c635c6034f3cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18e64090fe264086830c635c6034f3cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45653 } }
I20260928 02:38:28.335024 21650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18e64090fe264086830c635c6034f3cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18e64090fe264086830c635c6034f3cb. Latest consensus state: current_term: 1 leader_uuid: "18e64090fe264086830c635c6034f3cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18e64090fe264086830c635c6034f3cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45653 } } }
I20260928 02:38:28.335139 21649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18e64090fe264086830c635c6034f3cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18e64090fe264086830c635c6034f3cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18e64090fe264086830c635c6034f3cb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45653 } } }
I20260928 02:38:28.335173 21650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18e64090fe264086830c635c6034f3cb [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:28.335222 21649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18e64090fe264086830c635c6034f3cb [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:28.335670 21657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:38:28.335767 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 21577
I20260928 02:38:28.335870 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:38:28.336042 392 external_mini_cluster.cc:1428] Setting key 463a7f5e8bbed277d8229f074d85faec
I20260928 02:38:28.336150 21657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:38:28.337793 21657 catalog_manager.cc:1347] Generated new cluster ID: 50491d756e5447ee9def2d2f45bf57a1
I20260928 02:38:28.337842 21657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:38:28 dist-test-slave-fz47 krb5kdc[21564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:38:28.344606 21657 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:38:28.345304 21657 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:38:28.346966 21667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:28.336626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.64:60247 (local address 127.0.0.1:45653)
0928 02:38:28.336780 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:28.336788 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:28.336816 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:38:28.337286 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:28.337294 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:28.337350 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:28.337487 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:28.338213 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:28.339599 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:28.340576 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:28.341189 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:28.341319 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:28.343858 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:28.343862 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:28.343862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:28.343874 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:28.345573 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:28.345902 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:28.345905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:28.345905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:28.345993 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:28.346211 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:28.346212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:28.346212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:28.346425 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:28.346588 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:28.346667 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 02:38:28.346758 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260928 02:38:28.347772 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:38:28.351274 21657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18e64090fe264086830c635c6034f3cb: Generated new TSK 0
I20260928 02:38:28.351423 21657 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:38:28.362622 21667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:28.349544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.64:55241 (local address 127.0.0.1:45653)
0928 02:38:28.349575 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:28.349577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:28.357414 (+ 7837us) server_negotiation.cc:407] Connection header received
0928 02:38:28.357470 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:28.357470 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:28.357479 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:28.357522 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:28.357973 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:28.358453 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:28.359128 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:28.359350 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:28.359402 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:28.360417 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:28.360418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:28.360418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:28.360425 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:28.361640 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:28.361959 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:28.361960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:28.361960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:28.362008 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:28.362190 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:28.362190 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:28.362191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:28.362284 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:28.362382 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:28.362476 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 02:38:28.362531 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 02:38:28.364231 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 21577
W20260928 02:38:28.367898 21562 connection.cc:535] client connection to 127.0.0.1:45653 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45653): Connection reset by peer (error 104)
2026-09-28T02:38:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1206 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:38:28 dist-test-slave-fz47 krb5kdc[21676](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:38:28 dist-test-slave-fz47 krb5kdc[21676](info): set up 2 sockets
Sep 28 02:38:28 dist-test-slave-fz47 krb5kdc[21676](info): commencing operation
krb5kdc: starting...
W20260928 02:38:29.313163 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:38:29 dist-test-slave-fz47 krb5kdc[21676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:38:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:38:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:38:29.427883 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37419
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:29.505438 21689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:29.505709 21689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:29.505743 21689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:29.507419 21689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:29.507474 21689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:29.507493 21689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:29.507510 21689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:29.507534 21689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:29.509640 21689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37419
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:29.509907 21689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:29.510141 21689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:29.515471 21694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:29.515486 21695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:29.515468 21697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:29.515815 21689 server_base.cc:1034] running on GCE node
I20260928 02:38:29.516148 21689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:29.516515 21689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:29.517741 21689 hybrid_clock.cc:648] HybridClock initialized: now 1790563109517723 us; error 32 us; skew 500 ppm
Sep 28 02:38:29 dist-test-slave-fz47 krb5kdc[21676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:29.520727 21689 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:38:29.521162 21689 webserver.cc:460] Webserver started at http://127.0.0.1:41957/ using document root <none> and password file <none>
I20260928 02:38:29.521422 21689 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:29.521482 21689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:29.521610 21689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:29.522662 21689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "f07b0d5e689a4d8d89b1aad2f70d7eba"
format_stamp: "Formatted at 2026-09-28 02:38:29 on dist-test-slave-fz47"
server_key: "2a359ba8139a4ccb0cde2313f0278072"
server_key_iv: "2bf59df167d523b861744d2b35d87a4e"
server_key_version: "encryptionkey@0"
I20260928 02:38:29.522979 21689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "f07b0d5e689a4d8d89b1aad2f70d7eba"
format_stamp: "Formatted at 2026-09-28 02:38:29 on dist-test-slave-fz47"
server_key: "2a359ba8139a4ccb0cde2313f0278072"
server_key_iv: "2bf59df167d523b861744d2b35d87a4e"
server_key_version: "encryptionkey@0"
I20260928 02:38:29.524205 21689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:38:29.524866 21704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:29.525025 21689 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:29.525101 21689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "f07b0d5e689a4d8d89b1aad2f70d7eba"
format_stamp: "Formatted at 2026-09-28 02:38:29 on dist-test-slave-fz47"
server_key: "2a359ba8139a4ccb0cde2313f0278072"
server_key_iv: "2bf59df167d523b861744d2b35d87a4e"
server_key_version: "encryptionkey@0"
I20260928 02:38:29.525161 21689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:29.538271 21689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:29.539834 21689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:29.539958 21689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:29.543588 21689 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37419
I20260928 02:38:29.543618 21756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37419 every 8 connection(s)
I20260928 02:38:29.543916 21689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:38:29.544842 21757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:29.547132 21757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f07b0d5e689a4d8d89b1aad2f70d7eba: Bootstrap starting.
I20260928 02:38:29.547674 21757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f07b0d5e689a4d8d89b1aad2f70d7eba: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:29.547919 21757 log.cc:824] T 00000000000000000000000000000000 P f07b0d5e689a4d8d89b1aad2f70d7eba: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:29.548585 21757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f07b0d5e689a4d8d89b1aad2f70d7eba: No bootstrap required, opened a new log
I20260928 02:38:29.550213 21757 raft_consensus.cc:348] T 00000000000000000000000000000000 P f07b0d5e689a4d8d89b1aad2f70d7eba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07b0d5e689a4d8d89b1aad2f70d7eba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37419 } }
I20260928 02:38:29.550349 21757 raft_consensus.cc:374] T 00000000000000000000000000000000 P f07b0d5e689a4d8d89b1aad2f70d7eba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:29.550390 21757 raft_consensus.cc:729] T 00000000000000000000000000000000 P f07b0d5e689a4d8d89b1aad2f70d7eba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f07b0d5e689a4d8d89b1aad2f70d7eba, State: Initialized, Role: FOLLOWER
I20260928 02:38:29.550529 21757 consensus_queue.cc:260] T 00000000000000000000000000000000 P f07b0d5e689a4d8d89b1aad2f70d7eba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07b0d5e689a4d8d89b1aad2f70d7eba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37419 } }
I20260928 02:38:29.550586 21757 raft_consensus.cc:388] T 00000000000000000000000000000000 P f07b0d5e689a4d8d89b1aad2f70d7eba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:38:29.550607 21757 raft_consensus.cc:482] T 00000000000000000000000000000000 P f07b0d5e689a4d8d89b1aad2f70d7eba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:38:29.550661 21757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f07b0d5e689a4d8d89b1aad2f70d7eba [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:29.551162 21757 raft_consensus.cc:504] T 00000000000000000000000000000000 P f07b0d5e689a4d8d89b1aad2f70d7eba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07b0d5e689a4d8d89b1aad2f70d7eba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37419 } }
I20260928 02:38:29.551254 21757 leader_election.cc:302] T 00000000000000000000000000000000 P f07b0d5e689a4d8d89b1aad2f70d7eba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f07b0d5e689a4d8d89b1aad2f70d7eba; no voters:
I20260928 02:38:29.551414 21757 leader_election.cc:288] T 00000000000000000000000000000000 P f07b0d5e689a4d8d89b1aad2f70d7eba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:38:29.551457 21760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f07b0d5e689a4d8d89b1aad2f70d7eba [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:29.551606 21760 raft_consensus.cc:686] T 00000000000000000000000000000000 P f07b0d5e689a4d8d89b1aad2f70d7eba [term 1 LEADER]: Becoming Leader. State: Replica: f07b0d5e689a4d8d89b1aad2f70d7eba, State: Running, Role: LEADER
I20260928 02:38:29.551653 21757 sys_catalog.cc:564] T 00000000000000000000000000000000 P f07b0d5e689a4d8d89b1aad2f70d7eba [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:38:29.551715 21760 consensus_queue.cc:237] T 00000000000000000000000000000000 P f07b0d5e689a4d8d89b1aad2f70d7eba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07b0d5e689a4d8d89b1aad2f70d7eba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37419 } }
I20260928 02:38:29.551738 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 21689
I20260928 02:38:29.551842 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:38:29.552039 392 external_mini_cluster.cc:1428] Setting key 001fb18239b066e126f40939da0daa58
I20260928 02:38:29.552131 21761 sys_catalog.cc:455] T 00000000000000000000000000000000 P f07b0d5e689a4d8d89b1aad2f70d7eba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f07b0d5e689a4d8d89b1aad2f70d7eba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07b0d5e689a4d8d89b1aad2f70d7eba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37419 } } }
I20260928 02:38:29.552201 21761 sys_catalog.cc:458] T 00000000000000000000000000000000 P f07b0d5e689a4d8d89b1aad2f70d7eba [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:29.552258 21762 sys_catalog.cc:455] T 00000000000000000000000000000000 P f07b0d5e689a4d8d89b1aad2f70d7eba [sys.catalog]: SysCatalogTable state changed. Reason: New leader f07b0d5e689a4d8d89b1aad2f70d7eba. Latest consensus state: current_term: 1 leader_uuid: "f07b0d5e689a4d8d89b1aad2f70d7eba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f07b0d5e689a4d8d89b1aad2f70d7eba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37419 } } }
I20260928 02:38:29.552304 21762 sys_catalog.cc:458] T 00000000000000000000000000000000 P f07b0d5e689a4d8d89b1aad2f70d7eba [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:29.552945 21768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:38:29.553675 21768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:38:29.557225 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:29.552506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.64:47339 (local address 127.0.0.1:37419)
0928 02:38:29.552786 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:29.552796 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:29.552916 (+ 120us) server_negotiation.cc:407] Connection header received
0928 02:38:29.553126 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:29.553134 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:29.553202 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:29.553369 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:29.553578 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:29.554683 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:29.555437 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:29.555861 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:29.555951 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:29.556647 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:29.556651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:29.556652 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:29.556817 (+ 165us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:29.556845 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:29.556856 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:29.556922 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:29.557038 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 02:38:29.557066 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":79,"threads_started":1}
I20260928 02:38:29.557680 21768 catalog_manager.cc:1347] Generated new cluster ID: e8bd750f5a1b4bd6a27c9b2caa118191
I20260928 02:38:29.557725 21768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:38:29.569913 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:38:29.570513 21768 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:38:29.571157 21768 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:38:29.575054 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:29.571398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.64:48567 (local address 127.0.0.1:37419)
0928 02:38:29.571444 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:29.571446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:29.571546 (+ 100us) server_negotiation.cc:407] Connection header received
0928 02:38:29.571602 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:29.571603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:29.571613 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:29.571668 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:29.571790 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:29.572550 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:29.573663 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:29.574001 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:29.574039 (+ 38us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:29.574785 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:29.574787 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:29.574787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:29.574792 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:29.574807 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:29.574812 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:29.574857 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:29.574963 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 02:38:29.574975 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 02:38:29.576205 21768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f07b0d5e689a4d8d89b1aad2f70d7eba: Generated new TSK 0
I20260928 02:38:29.576359 21768 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:38:29.578487 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 21689
W20260928 02:38:29.582427 21674 connection.cc:535] client connection to 127.0.0.1:37419 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37419): Connection reset by peer (error 104)
2026-09-28T02:38:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1214 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:38:29 dist-test-slave-fz47 krb5kdc[21788](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:38:29 dist-test-slave-fz47 krb5kdc[21788](info): set up 2 sockets
Sep 28 02:38:29 dist-test-slave-fz47 krb5kdc[21788](info): commencing operation
krb5kdc: starting...
W20260928 02:38:30.533079 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:38:30 dist-test-slave-fz47 krb5kdc[21788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:38:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:38:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:38:30.641464 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40249
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:30.719015 21801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:30.719203 21801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:30.719233 21801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:30.720330 21801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:30.720368 21801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:30.720386 21801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:30.720402 21801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:30.720418 21801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:30.721818 21801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40249
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790562862804661-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:30.722054 21801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:30.722244 21801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:30.726768 21807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:30.726742 21806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:30.726811 21801 server_base.cc:1034] running on GCE node
W20260928 02:38:30.726754 21809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:30.727173 21801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:30.727456 21801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:30.728600 21801 hybrid_clock.cc:648] HybridClock initialized: now 1790563110728585 us; error 29 us; skew 500 ppm
Sep 28 02:38:30 dist-test-slave-fz47 krb5kdc[21788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:30.738117 21801 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:38:30.738626 21801 webserver.cc:460] Webserver started at http://127.0.0.1:44851/ using document root <none> and password file <none>
I20260928 02:38:30.738840 21801 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:30.738889 21801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:30.739022 21801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:30.740083 21801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "59d0a18fc1ab47d2ac50036a8410dbd4"
format_stamp: "Formatted at 2026-09-28 02:38:30 on dist-test-slave-fz47"
server_key: "e470aa0b9227822fb1b0b849a33c14f9"
server_key_iv: "53ca9b34d0c7a7c8e12bdacf4e49d2ae"
server_key_version: "encryptionkey@0"
I20260928 02:38:30.740410 21801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "59d0a18fc1ab47d2ac50036a8410dbd4"
format_stamp: "Formatted at 2026-09-28 02:38:30 on dist-test-slave-fz47"
server_key: "e470aa0b9227822fb1b0b849a33c14f9"
server_key_iv: "53ca9b34d0c7a7c8e12bdacf4e49d2ae"
server_key_version: "encryptionkey@0"
I20260928 02:38:30.741739 21801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:38:30.742400 21816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:30.742556 21801 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:30.742637 21801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "59d0a18fc1ab47d2ac50036a8410dbd4"
format_stamp: "Formatted at 2026-09-28 02:38:30 on dist-test-slave-fz47"
server_key: "e470aa0b9227822fb1b0b849a33c14f9"
server_key_iv: "53ca9b34d0c7a7c8e12bdacf4e49d2ae"
server_key_version: "encryptionkey@0"
I20260928 02:38:30.742697 21801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:30.745906 21801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:30.747651 21801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:30.747772 21801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:30.751396 21801 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40249
I20260928 02:38:30.751439 21868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40249 every 8 connection(s)
I20260928 02:38:30.751734 21801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:38:30.752669 21869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:30.754823 21869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59d0a18fc1ab47d2ac50036a8410dbd4: Bootstrap starting.
I20260928 02:38:30.755323 21869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59d0a18fc1ab47d2ac50036a8410dbd4: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:30.755554 21869 log.cc:824] T 00000000000000000000000000000000 P 59d0a18fc1ab47d2ac50036a8410dbd4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:30.755689 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 21801
I20260928 02:38:30.755785 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:38:30.755947 392 external_mini_cluster.cc:1428] Setting key ce5a8021b80da8059b9a926389163ed3
I20260928 02:38:30.756192 21869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59d0a18fc1ab47d2ac50036a8410dbd4: No bootstrap required, opened a new log
I20260928 02:38:30.758383 21869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59d0a18fc1ab47d2ac50036a8410dbd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d0a18fc1ab47d2ac50036a8410dbd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40249 } }
I20260928 02:38:30.758548 21869 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59d0a18fc1ab47d2ac50036a8410dbd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:30.758606 21869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59d0a18fc1ab47d2ac50036a8410dbd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59d0a18fc1ab47d2ac50036a8410dbd4, State: Initialized, Role: FOLLOWER
I20260928 02:38:30.758725 21869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59d0a18fc1ab47d2ac50036a8410dbd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d0a18fc1ab47d2ac50036a8410dbd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40249 } }
I20260928 02:38:30.758793 21869 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59d0a18fc1ab47d2ac50036a8410dbd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:38:30.758822 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:30.756375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.64:48325 (local address 127.0.0.1:40249)
0928 02:38:30.756547 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:30.756555 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:30.756579 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:38:30.757517 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:30.757524 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:30.757550 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:30.757660 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:30.758240 (+ 580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:30.758243 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:30.758243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:30.758247 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:30.758281 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:30.758291 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:30.758339 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:30.758425 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 02:38:30.758459 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260928 02:38:30.758823 21869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59d0a18fc1ab47d2ac50036a8410dbd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:38:30.758913 21869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59d0a18fc1ab47d2ac50036a8410dbd4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:30.759475 21869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59d0a18fc1ab47d2ac50036a8410dbd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d0a18fc1ab47d2ac50036a8410dbd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40249 } }
I20260928 02:38:30.759568 21869 leader_election.cc:302] T 00000000000000000000000000000000 P 59d0a18fc1ab47d2ac50036a8410dbd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59d0a18fc1ab47d2ac50036a8410dbd4; no voters:
I20260928 02:38:30.759709 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:38:30.759722 21869 leader_election.cc:288] T 00000000000000000000000000000000 P 59d0a18fc1ab47d2ac50036a8410dbd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:38:30.759749 21874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59d0a18fc1ab47d2ac50036a8410dbd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:30.759860 21874 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59d0a18fc1ab47d2ac50036a8410dbd4 [term 1 LEADER]: Becoming Leader. State: Replica: 59d0a18fc1ab47d2ac50036a8410dbd4, State: Running, Role: LEADER
I20260928 02:38:30.759945 21869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59d0a18fc1ab47d2ac50036a8410dbd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:38:30.759994 21874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59d0a18fc1ab47d2ac50036a8410dbd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d0a18fc1ab47d2ac50036a8410dbd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40249 } }
I20260928 02:38:30.760334 21874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59d0a18fc1ab47d2ac50036a8410dbd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59d0a18fc1ab47d2ac50036a8410dbd4. Latest consensus state: current_term: 1 leader_uuid: "59d0a18fc1ab47d2ac50036a8410dbd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d0a18fc1ab47d2ac50036a8410dbd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40249 } } }
I20260928 02:38:30.760388 21876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59d0a18fc1ab47d2ac50036a8410dbd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59d0a18fc1ab47d2ac50036a8410dbd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d0a18fc1ab47d2ac50036a8410dbd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40249 } } }
I20260928 02:38:30.760481 21876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59d0a18fc1ab47d2ac50036a8410dbd4 [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:30.760464 21874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59d0a18fc1ab47d2ac50036a8410dbd4 [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:30.761243 21883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:38:30.761708 21883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:38:30.762267 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:30.761238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.64:47139 (local address 127.0.0.1:40249)
0928 02:38:30.761291 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:30.761293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:30.761456 (+ 163us) server_negotiation.cc:407] Connection header received
0928 02:38:30.761497 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:30.761498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:30.761503 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:30.761550 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:30.762086 (+ 536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:30.762087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:30.762088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:30.762091 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:30.762105 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:30.762109 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:30.762162 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:30.762203 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 02:38:30.762214 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:38:30.763202 21883 catalog_manager.cc:1347] Generated new cluster ID: 47d66db8cced4d388f4de4c851e46b0d
I20260928 02:38:30.763252 21883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:38:30.773324 21883 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:38:30.773478 21883 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:38:30.790854 21883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59d0a18fc1ab47d2ac50036a8410dbd4: Generated new TSK 0
I20260928 02:38:30.791067 21883 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:38:30.797761 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 21801
2026-09-28T02:38:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1218 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:38:30 dist-test-slave-fz47 krb5kdc[21900](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:38:30 dist-test-slave-fz47 krb5kdc[21900](info): set up 2 sockets
Sep 28 02:38:30 dist-test-slave-fz47 krb5kdc[21900](info): commencing operation
krb5kdc: starting...
W20260928 02:38:31.745042 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:38:31 dist-test-slave-fz47 krb5kdc[21900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:38:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:38:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 02:38:31.853912 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46741
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:31.932085 21913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:31.932277 21913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:31.932307 21913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:31.933449 21913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:31.933486 21913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:31.933504 21913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:31.933521 21913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:31.933535 21913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:31.934955 21913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46741
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:31.935182 21913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:31.935370 21913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:31.939836 21919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:31.939867 21921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:31.939885 21918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:31.939906 21913 server_base.cc:1034] running on GCE node
I20260928 02:38:31.940228 21913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:31.940485 21913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:31.941620 21913 hybrid_clock.cc:648] HybridClock initialized: now 1790563111941607 us; error 31 us; skew 500 ppm
Sep 28 02:38:31 dist-test-slave-fz47 krb5kdc[21900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:31.943861 21913 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:38:31.944254 21913 webserver.cc:460] Webserver started at http://127.0.98.62:40859/ using document root <none> and password file <none>
I20260928 02:38:31.944437 21913 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:31.944480 21913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:31.944597 21913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:31.945621 21913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "cefdb325d3534f7da2cefaa3ab4db4ca"
format_stamp: "Formatted at 2026-09-28 02:38:31 on dist-test-slave-fz47"
server_key: "629cab74482f2a6d14f6121ba2ce3df8"
server_key_iv: "a3ae52283d88dce3e9d07dff46e0dda5"
server_key_version: "encryptionkey@0"
I20260928 02:38:31.945938 21913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "cefdb325d3534f7da2cefaa3ab4db4ca"
format_stamp: "Formatted at 2026-09-28 02:38:31 on dist-test-slave-fz47"
server_key: "629cab74482f2a6d14f6121ba2ce3df8"
server_key_iv: "a3ae52283d88dce3e9d07dff46e0dda5"
server_key_version: "encryptionkey@0"
I20260928 02:38:31.947103 21913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:38:31.947757 21928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:31.947912 21913 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:31.947991 21913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "cefdb325d3534f7da2cefaa3ab4db4ca"
format_stamp: "Formatted at 2026-09-28 02:38:31 on dist-test-slave-fz47"
server_key: "629cab74482f2a6d14f6121ba2ce3df8"
server_key_iv: "a3ae52283d88dce3e9d07dff46e0dda5"
server_key_version: "encryptionkey@0"
I20260928 02:38:31.948050 21913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:31.958290 21913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:31.959918 21913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:31.960045 21913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:31.963676 21913 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46741
I20260928 02:38:31.963706 21980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46741 every 8 connection(s)
I20260928 02:38:31.964011 21913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:38:31.964794 21981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:31.967301 21981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cefdb325d3534f7da2cefaa3ab4db4ca: Bootstrap starting.
I20260928 02:38:31.967800 21981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cefdb325d3534f7da2cefaa3ab4db4ca: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:31.967985 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 21913
I20260928 02:38:31.968091 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:38:31.968072 21981 log.cc:824] T 00000000000000000000000000000000 P cefdb325d3534f7da2cefaa3ab4db4ca: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:31.968252 392 external_mini_cluster.cc:1428] Setting key 48b6815e620500473edc383188e417d2
I20260928 02:38:31.968779 21981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cefdb325d3534f7da2cefaa3ab4db4ca: No bootstrap required, opened a new log
I20260928 02:38:31.970577 21981 raft_consensus.cc:348] T 00000000000000000000000000000000 P cefdb325d3534f7da2cefaa3ab4db4ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cefdb325d3534f7da2cefaa3ab4db4ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46741 } }
I20260928 02:38:31.970707 21981 raft_consensus.cc:374] T 00000000000000000000000000000000 P cefdb325d3534f7da2cefaa3ab4db4ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:31.970733 21981 raft_consensus.cc:729] T 00000000000000000000000000000000 P cefdb325d3534f7da2cefaa3ab4db4ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cefdb325d3534f7da2cefaa3ab4db4ca, State: Initialized, Role: FOLLOWER
I20260928 02:38:31.970850 21981 consensus_queue.cc:260] T 00000000000000000000000000000000 P cefdb325d3534f7da2cefaa3ab4db4ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cefdb325d3534f7da2cefaa3ab4db4ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46741 } }
I20260928 02:38:31.970906 21981 raft_consensus.cc:388] T 00000000000000000000000000000000 P cefdb325d3534f7da2cefaa3ab4db4ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:38:31.970927 21981 raft_consensus.cc:482] T 00000000000000000000000000000000 P cefdb325d3534f7da2cefaa3ab4db4ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:38:31.970957 21981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cefdb325d3534f7da2cefaa3ab4db4ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:31.971467 21981 raft_consensus.cc:504] T 00000000000000000000000000000000 P cefdb325d3534f7da2cefaa3ab4db4ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cefdb325d3534f7da2cefaa3ab4db4ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46741 } }
I20260928 02:38:31.971549 21981 leader_election.cc:302] T 00000000000000000000000000000000 P cefdb325d3534f7da2cefaa3ab4db4ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cefdb325d3534f7da2cefaa3ab4db4ca; no voters:
I20260928 02:38:31.971689 21981 leader_election.cc:288] T 00000000000000000000000000000000 P cefdb325d3534f7da2cefaa3ab4db4ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:38:31.971730 21986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cefdb325d3534f7da2cefaa3ab4db4ca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:31.971874 21986 raft_consensus.cc:686] T 00000000000000000000000000000000 P cefdb325d3534f7da2cefaa3ab4db4ca [term 1 LEADER]: Becoming Leader. State: Replica: cefdb325d3534f7da2cefaa3ab4db4ca, State: Running, Role: LEADER
I20260928 02:38:31.971917 21981 sys_catalog.cc:564] T 00000000000000000000000000000000 P cefdb325d3534f7da2cefaa3ab4db4ca [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:38:31.971951 21986 consensus_queue.cc:237] T 00000000000000000000000000000000 P cefdb325d3534f7da2cefaa3ab4db4ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cefdb325d3534f7da2cefaa3ab4db4ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46741 } }
I20260928 02:38:31.972249 21987 sys_catalog.cc:455] T 00000000000000000000000000000000 P cefdb325d3534f7da2cefaa3ab4db4ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cefdb325d3534f7da2cefaa3ab4db4ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cefdb325d3534f7da2cefaa3ab4db4ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46741 } } }
I20260928 02:38:31.972457 21987 sys_catalog.cc:458] T 00000000000000000000000000000000 P cefdb325d3534f7da2cefaa3ab4db4ca [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:31.972246 21988 sys_catalog.cc:455] T 00000000000000000000000000000000 P cefdb325d3534f7da2cefaa3ab4db4ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader cefdb325d3534f7da2cefaa3ab4db4ca. Latest consensus state: current_term: 1 leader_uuid: "cefdb325d3534f7da2cefaa3ab4db4ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cefdb325d3534f7da2cefaa3ab4db4ca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46741 } } }
I20260928 02:38:31.972694 21988 sys_catalog.cc:458] T 00000000000000000000000000000000 P cefdb325d3534f7da2cefaa3ab4db4ca [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:31.972750 21995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:38:31.973207 21995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:38:31.974774 21995 catalog_manager.cc:1347] Generated new cluster ID: 6a167e76a5644142aaf8e351b4e94a78
I20260928 02:38:31.974823 21995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 02:38:31 dist-test-slave-fz47 krb5kdc[21900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:38:31.980013 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:31.968733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42889 (local address 127.0.98.62:46741)
0928 02:38:31.968924 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:31.968934 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:31.968961 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:38:31.969920 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:31.969927 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:31.969981 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:31.970101 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:31.970684 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:31.972056 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:31.973123 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:31.973687 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:31.973793 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:31.976601 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:31.976605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:31.976606 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:31.976617 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:31.978436 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:31.978888 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:31.978890 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:31.978891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:31.978985 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:31.979201 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:31.979202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:31.979209 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:31.979471 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:31.979601 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:31.979743 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 02:38:31.979832 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":81,"threads_started":1}
I20260928 02:38:31.980911 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:38:31.989064 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:31.982374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35273 (local address 127.0.98.62:46741)
0928 02:38:31.982417 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:31.982419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:31.982518 (+ 99us) server_negotiation.cc:407] Connection header received
0928 02:38:31.982560 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:31.982561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:31.982570 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:31.982635 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:31.983164 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:31.983933 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:31.984852 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:31.985184 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:31.985249 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:31.986515 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:31.986516 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:31.986516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:31.986523 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:31.987979 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:31.988357 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:31.988358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:31.988359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:31.988415 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:31.988597 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:31.988597 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:31.988598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:31.988720 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:31.988814 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:31.988911 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 02:38:31.988970 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":23}
I20260928 02:38:31.995749 21995 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:38:31.996292 21995 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:38:32.000662 21995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cefdb325d3534f7da2cefaa3ab4db4ca: Generated new TSK 0
I20260928 02:38:32.000836 21995 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:38:32.003309 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 21913
W20260928 02:38:32.006994 21898 connection.cc:535] client connection to 127.0.98.62:46741 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:46741): Connection reset by peer (error 104)
2026-09-28T02:38:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1205 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:38:32 dist-test-slave-fz47 krb5kdc[22012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:38:32 dist-test-slave-fz47 krb5kdc[22012](info): set up 2 sockets
Sep 28 02:38:32 dist-test-slave-fz47 krb5kdc[22012](info): commencing operation
krb5kdc: starting...
W20260928 02:38:32.944981 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:38:33 dist-test-slave-fz47 krb5kdc[22012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:38:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:38:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 02:38:33.048132 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40667
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:33.126235 22025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:33.126431 22025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:33.126452 22025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:33.127575 22025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:33.127616 22025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:33.127628 22025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:33.127640 22025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:33.127651 22025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:33.129066 22025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40667
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:33.129298 22025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:33.129513 22025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:33.134027 22030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:33.134048 22033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:33.134269 22025 server_base.cc:1034] running on GCE node
W20260928 02:38:33.134056 22031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:33.134521 22025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:33.134802 22025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:33.135953 22025 hybrid_clock.cc:648] HybridClock initialized: now 1790563113135941 us; error 35 us; skew 500 ppm
Sep 28 02:38:33 dist-test-slave-fz47 krb5kdc[22012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:33.138357 22025 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:38:33.138777 22025 webserver.cc:460] Webserver started at http://127.0.98.62:41705/ using document root <none> and password file <none>
I20260928 02:38:33.138976 22025 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:33.139021 22025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:33.139143 22025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:33.140149 22025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "b23e4d6c34b841efba360085385f2c76"
format_stamp: "Formatted at 2026-09-28 02:38:33 on dist-test-slave-fz47"
server_key: "a8d3544266cdfc5482ebee3804409a0d"
server_key_iv: "5003b9620902b4424190523368af320e"
server_key_version: "encryptionkey@0"
I20260928 02:38:33.140460 22025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "b23e4d6c34b841efba360085385f2c76"
format_stamp: "Formatted at 2026-09-28 02:38:33 on dist-test-slave-fz47"
server_key: "a8d3544266cdfc5482ebee3804409a0d"
server_key_iv: "5003b9620902b4424190523368af320e"
server_key_version: "encryptionkey@0"
I20260928 02:38:33.141752 22025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:38:33.142467 22040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:33.142625 22025 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:33.142705 22025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "b23e4d6c34b841efba360085385f2c76"
format_stamp: "Formatted at 2026-09-28 02:38:33 on dist-test-slave-fz47"
server_key: "a8d3544266cdfc5482ebee3804409a0d"
server_key_iv: "5003b9620902b4424190523368af320e"
server_key_version: "encryptionkey@0"
I20260928 02:38:33.142765 22025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:33.150091 22025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:33.151995 22025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:33.152120 22025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:33.155699 22025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40667
I20260928 02:38:33.155733 22092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40667 every 8 connection(s)
I20260928 02:38:33.156024 22025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:38:33.156796 22093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:33.158941 22093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b23e4d6c34b841efba360085385f2c76: Bootstrap starting.
I20260928 02:38:33.159452 22093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b23e4d6c34b841efba360085385f2c76: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:33.159688 22093 log.cc:824] T 00000000000000000000000000000000 P b23e4d6c34b841efba360085385f2c76: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:33.160297 22093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b23e4d6c34b841efba360085385f2c76: No bootstrap required, opened a new log
I20260928 02:38:33.161830 22093 raft_consensus.cc:348] T 00000000000000000000000000000000 P b23e4d6c34b841efba360085385f2c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b23e4d6c34b841efba360085385f2c76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40667 } }
I20260928 02:38:33.162003 22093 raft_consensus.cc:374] T 00000000000000000000000000000000 P b23e4d6c34b841efba360085385f2c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:33.162025 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 22025
I20260928 02:38:33.162036 22093 raft_consensus.cc:729] T 00000000000000000000000000000000 P b23e4d6c34b841efba360085385f2c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b23e4d6c34b841efba360085385f2c76, State: Initialized, Role: FOLLOWER
I20260928 02:38:33.162135 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:38:33.162177 22093 consensus_queue.cc:260] T 00000000000000000000000000000000 P b23e4d6c34b841efba360085385f2c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b23e4d6c34b841efba360085385f2c76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40667 } }
I20260928 02:38:33.162266 22093 raft_consensus.cc:388] T 00000000000000000000000000000000 P b23e4d6c34b841efba360085385f2c76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:38:33.162302 22093 raft_consensus.cc:482] T 00000000000000000000000000000000 P b23e4d6c34b841efba360085385f2c76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:38:33.162313 392 external_mini_cluster.cc:1428] Setting key 82f97e684ce7d67ea8c1c4122e6ab027
I20260928 02:38:33.162348 22093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b23e4d6c34b841efba360085385f2c76 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:33.162947 22093 raft_consensus.cc:504] T 00000000000000000000000000000000 P b23e4d6c34b841efba360085385f2c76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b23e4d6c34b841efba360085385f2c76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40667 } }
I20260928 02:38:33.163043 22093 leader_election.cc:302] T 00000000000000000000000000000000 P b23e4d6c34b841efba360085385f2c76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b23e4d6c34b841efba360085385f2c76; no voters:
I20260928 02:38:33.163224 22093 leader_election.cc:288] T 00000000000000000000000000000000 P b23e4d6c34b841efba360085385f2c76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:38:33.163251 22098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b23e4d6c34b841efba360085385f2c76 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:33.163375 22098 raft_consensus.cc:686] T 00000000000000000000000000000000 P b23e4d6c34b841efba360085385f2c76 [term 1 LEADER]: Becoming Leader. State: Replica: b23e4d6c34b841efba360085385f2c76, State: Running, Role: LEADER
I20260928 02:38:33.163457 22093 sys_catalog.cc:564] T 00000000000000000000000000000000 P b23e4d6c34b841efba360085385f2c76 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:38:33.163496 22098 consensus_queue.cc:237] T 00000000000000000000000000000000 P b23e4d6c34b841efba360085385f2c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b23e4d6c34b841efba360085385f2c76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40667 } }
I20260928 02:38:33.163893 22099 sys_catalog.cc:455] T 00000000000000000000000000000000 P b23e4d6c34b841efba360085385f2c76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b23e4d6c34b841efba360085385f2c76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b23e4d6c34b841efba360085385f2c76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40667 } } }
I20260928 02:38:33.164032 22099 sys_catalog.cc:458] T 00000000000000000000000000000000 P b23e4d6c34b841efba360085385f2c76 [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:33.163878 22098 sys_catalog.cc:455] T 00000000000000000000000000000000 P b23e4d6c34b841efba360085385f2c76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b23e4d6c34b841efba360085385f2c76. Latest consensus state: current_term: 1 leader_uuid: "b23e4d6c34b841efba360085385f2c76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b23e4d6c34b841efba360085385f2c76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40667 } } }
I20260928 02:38:33.164542 22107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:38:33.164832 22098 sys_catalog.cc:458] T 00000000000000000000000000000000 P b23e4d6c34b841efba360085385f2c76 [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:33.165198 22107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:38:33.166642 22107 catalog_manager.cc:1347] Generated new cluster ID: 914825dda8bd4e81811b213f566d6dc7
I20260928 02:38:33.166688 22107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:38:33.168258 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:33.162986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56227 (local address 127.0.98.62:40667)
0928 02:38:33.163136 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:33.163145 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:33.163178 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:38:33.163303 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:33.163312 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:33.163367 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:33.163501 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:33.163744 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:33.165385 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:33.166266 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:33.166829 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:33.166935 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:33.167586 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:33.167590 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:33.167590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:33.167788 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:33.167832 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:33.167847 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:33.167933 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:33.168003 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 02:38:33.168032 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":99,"thread_start_us":66,"threads_started":1}
I20260928 02:38:33.169076 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:38:33.176573 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:33.170398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49211 (local address 127.0.98.62:40667)
0928 02:38:33.170427 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:33.170429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:33.173394 (+ 2965us) server_negotiation.cc:407] Connection header received
0928 02:38:33.173441 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:33.173442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:33.173453 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:33.173510 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:33.173622 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:33.174384 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:33.175296 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:33.175627 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:33.175686 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:33.176342 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:33.176343 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:33.176344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:33.176348 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:33.176362 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:33.176367 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:33.176430 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:33.176489 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 02:38:33.176500 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 02:38:33.177565 22107 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:38:33.178020 22107 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:38:33.184077 22107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b23e4d6c34b841efba360085385f2c76: Generated new TSK 0
I20260928 02:38:33.184258 22107 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:38:33.188076 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 22025
W20260928 02:38:33.191932 22010 connection.cc:535] client connection to 127.0.98.62:40667 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:40667): Connection reset by peer (error 104)
2026-09-28T02:38:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1184 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:38:33 dist-test-slave-fz47 krb5kdc[22124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:38:33 dist-test-slave-fz47 krb5kdc[22124](info): set up 2 sockets
Sep 28 02:38:33 dist-test-slave-fz47 krb5kdc[22124](info): commencing operation
krb5kdc: starting...
W20260928 02:38:34.136981 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:38:34 dist-test-slave-fz47 krb5kdc[22124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:38:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:38:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 02:38:34.243530 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35865
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:34.321655 22137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:34.321880 22137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:34.321911 22137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:34.323038 22137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:34.323079 22137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:34.323098 22137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:34.323114 22137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:34.323128 22137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:34.324555 22137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35865
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:34.324781 22137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:34.324970 22137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:34.329483 22143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:34.329493 22145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:34.329598 22137 server_base.cc:1034] running on GCE node
W20260928 02:38:34.329542 22142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:34.329881 22137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:34.330152 22137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:34.331285 22137 hybrid_clock.cc:648] HybridClock initialized: now 1790563114331271 us; error 28 us; skew 500 ppm
Sep 28 02:38:34 dist-test-slave-fz47 krb5kdc[22124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:34.333621 22137 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:38:34.334019 22137 webserver.cc:460] Webserver started at http://127.0.98.62:36215/ using document root <none> and password file <none>
I20260928 02:38:34.334211 22137 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:34.334256 22137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:34.334372 22137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:34.335383 22137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "fccb7a7f0f404794b838cac8260dd681"
format_stamp: "Formatted at 2026-09-28 02:38:34 on dist-test-slave-fz47"
server_key: "d1c5312194b8e270bf3541fcfba2d598"
server_key_iv: "c6a49d59f1f2d4b7eed6b4ec1452db5b"
server_key_version: "encryptionkey@0"
I20260928 02:38:34.335702 22137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "fccb7a7f0f404794b838cac8260dd681"
format_stamp: "Formatted at 2026-09-28 02:38:34 on dist-test-slave-fz47"
server_key: "d1c5312194b8e270bf3541fcfba2d598"
server_key_iv: "c6a49d59f1f2d4b7eed6b4ec1452db5b"
server_key_version: "encryptionkey@0"
I20260928 02:38:34.336921 22137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:38:34.337627 22152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:34.337777 22137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:34.337855 22137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "fccb7a7f0f404794b838cac8260dd681"
format_stamp: "Formatted at 2026-09-28 02:38:34 on dist-test-slave-fz47"
server_key: "d1c5312194b8e270bf3541fcfba2d598"
server_key_iv: "c6a49d59f1f2d4b7eed6b4ec1452db5b"
server_key_version: "encryptionkey@0"
I20260928 02:38:34.337913 22137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:34.340917 22137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:34.342865 22137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:34.342988 22137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:34.346520 22137 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35865
I20260928 02:38:34.346549 22204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35865 every 8 connection(s)
I20260928 02:38:34.346812 22137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:38:34.347282 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 22137
I20260928 02:38:34.347414 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:38:34.347592 392 external_mini_cluster.cc:1428] Setting key fbef1b0bbe92c85a951f6bd6d188ffb2
I20260928 02:38:34.347822 22205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:34.349809 22208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:34.348046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40777 (local address 127.0.98.62:35865)
0928 02:38:34.348211 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:34.348218 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:34.348241 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:38:34.348752 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:34.348757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:34.348782 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:34.348885 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:34.349438 (+ 553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:34.349441 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:34.349441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:34.349445 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:34.349477 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:34.349487 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:34.349536 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:34.349611 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 02:38:34.349642 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":69,"threads_started":1}
I20260928 02:38:34.350682 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:38:34.350831 22205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fccb7a7f0f404794b838cac8260dd681: Bootstrap starting.
I20260928 02:38:34.351471 22205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fccb7a7f0f404794b838cac8260dd681: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:34.351778 22205 log.cc:824] T 00000000000000000000000000000000 P fccb7a7f0f404794b838cac8260dd681: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:34.352380 22205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fccb7a7f0f404794b838cac8260dd681: No bootstrap required, opened a new log
I20260928 02:38:34.354221 22208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:34.352125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54417 (local address 127.0.98.62:35865)
0928 02:38:34.352160 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:34.352161 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:34.353394 (+ 1233us) server_negotiation.cc:407] Connection header received
0928 02:38:34.353452 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:34.353453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:34.353457 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:34.353496 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:34.354027 (+ 531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:34.354028 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:34.354028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:34.354032 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:34.354052 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:34.354056 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:34.354102 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:34.354151 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 02:38:34.354164 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:38:34.354137 22205 raft_consensus.cc:348] T 00000000000000000000000000000000 P fccb7a7f0f404794b838cac8260dd681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fccb7a7f0f404794b838cac8260dd681" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35865 } }
I20260928 02:38:34.354290 22205 raft_consensus.cc:374] T 00000000000000000000000000000000 P fccb7a7f0f404794b838cac8260dd681 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:34.354331 22205 raft_consensus.cc:729] T 00000000000000000000000000000000 P fccb7a7f0f404794b838cac8260dd681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fccb7a7f0f404794b838cac8260dd681, State: Initialized, Role: FOLLOWER
I20260928 02:38:34.354441 22205 consensus_queue.cc:260] T 00000000000000000000000000000000 P fccb7a7f0f404794b838cac8260dd681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fccb7a7f0f404794b838cac8260dd681" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35865 } }
I20260928 02:38:34.354527 22205 raft_consensus.cc:388] T 00000000000000000000000000000000 P fccb7a7f0f404794b838cac8260dd681 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:38:34.354562 22205 raft_consensus.cc:482] T 00000000000000000000000000000000 P fccb7a7f0f404794b838cac8260dd681 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:38:34.354605 22205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fccb7a7f0f404794b838cac8260dd681 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:34.355067 22205 raft_consensus.cc:504] T 00000000000000000000000000000000 P fccb7a7f0f404794b838cac8260dd681 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fccb7a7f0f404794b838cac8260dd681" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35865 } }
I20260928 02:38:34.355136 22205 leader_election.cc:302] T 00000000000000000000000000000000 P fccb7a7f0f404794b838cac8260dd681 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fccb7a7f0f404794b838cac8260dd681; no voters:
I20260928 02:38:34.355255 22205 leader_election.cc:288] T 00000000000000000000000000000000 P fccb7a7f0f404794b838cac8260dd681 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:38:34.355304 22215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fccb7a7f0f404794b838cac8260dd681 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:34.355412 22215 raft_consensus.cc:686] T 00000000000000000000000000000000 P fccb7a7f0f404794b838cac8260dd681 [term 1 LEADER]: Becoming Leader. State: Replica: fccb7a7f0f404794b838cac8260dd681, State: Running, Role: LEADER
I20260928 02:38:34.355437 22205 sys_catalog.cc:564] T 00000000000000000000000000000000 P fccb7a7f0f404794b838cac8260dd681 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:38:34.355482 22215 consensus_queue.cc:237] T 00000000000000000000000000000000 P fccb7a7f0f404794b838cac8260dd681 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fccb7a7f0f404794b838cac8260dd681" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35865 } }
I20260928 02:38:34.355798 22216 sys_catalog.cc:455] T 00000000000000000000000000000000 P fccb7a7f0f404794b838cac8260dd681 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fccb7a7f0f404794b838cac8260dd681" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fccb7a7f0f404794b838cac8260dd681" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35865 } } }
I20260928 02:38:34.355809 22217 sys_catalog.cc:455] T 00000000000000000000000000000000 P fccb7a7f0f404794b838cac8260dd681 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fccb7a7f0f404794b838cac8260dd681. Latest consensus state: current_term: 1 leader_uuid: "fccb7a7f0f404794b838cac8260dd681" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fccb7a7f0f404794b838cac8260dd681" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35865 } } }
I20260928 02:38:34.355891 22216 sys_catalog.cc:458] T 00000000000000000000000000000000 P fccb7a7f0f404794b838cac8260dd681 [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:34.355895 22217 sys_catalog.cc:458] T 00000000000000000000000000000000 P fccb7a7f0f404794b838cac8260dd681 [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:34.356323 22223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:38:34.356758 22223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:38:34.358081 22223 catalog_manager.cc:1347] Generated new cluster ID: 90fe1b64c52848f5b2499e6a678021c3
I20260928 02:38:34.358127 22223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:38:34.380824 22223 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:38:34.380993 22223 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:38:34.386720 22223 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fccb7a7f0f404794b838cac8260dd681: Generated new TSK 0
I20260928 02:38:34.386924 22223 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:38:34.395589 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 22137
2026-09-28T02:38:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1207 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:38:34 dist-test-slave-fz47 krb5kdc[22236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:38:34 dist-test-slave-fz47 krb5kdc[22236](info): set up 2 sockets
Sep 28 02:38:34 dist-test-slave-fz47 krb5kdc[22236](info): commencing operation
krb5kdc: starting...
W20260928 02:38:35.344983 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:38:35 dist-test-slave-fz47 krb5kdc[22236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:38:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:38:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 02:38:35.457852 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34961
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:35.533989 22249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:35.534179 22249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:35.534209 22249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:35.535326 22249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:35.535368 22249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:35.535387 22249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:35.535403 22249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:35.535418 22249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:35.536888 22249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:35.537138 22249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:35.537353 22249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:35.541839 22255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:35.541865 22249 server_base.cc:1034] running on GCE node
W20260928 02:38:35.541873 22254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:35.541841 22257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:35.542260 22249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:35.542549 22249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:35.543712 22249 hybrid_clock.cc:648] HybridClock initialized: now 1790563115543690 us; error 30 us; skew 500 ppm
Sep 28 02:38:35 dist-test-slave-fz47 krb5kdc[22236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:35.546072 22249 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:38:35.546458 22249 webserver.cc:460] Webserver started at http://127.0.98.62:40215/ using document root <none> and password file <none>
I20260928 02:38:35.546653 22249 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:35.546698 22249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:35.546818 22249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:35.547813 22249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "6480d548d1d545bf8b24d7cbe0ce5eae"
format_stamp: "Formatted at 2026-09-28 02:38:35 on dist-test-slave-fz47"
server_key: "a5ae149fbeb423db27a16253d66b1da3"
server_key_iv: "a2249e7698c34f7924d3bb8e2d08e64d"
server_key_version: "encryptionkey@0"
I20260928 02:38:35.548135 22249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "6480d548d1d545bf8b24d7cbe0ce5eae"
format_stamp: "Formatted at 2026-09-28 02:38:35 on dist-test-slave-fz47"
server_key: "a5ae149fbeb423db27a16253d66b1da3"
server_key_iv: "a2249e7698c34f7924d3bb8e2d08e64d"
server_key_version: "encryptionkey@0"
I20260928 02:38:35.549373 22249 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:38:35.550143 22264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:35.550310 22249 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:35.550390 22249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "6480d548d1d545bf8b24d7cbe0ce5eae"
format_stamp: "Formatted at 2026-09-28 02:38:35 on dist-test-slave-fz47"
server_key: "a5ae149fbeb423db27a16253d66b1da3"
server_key_iv: "a2249e7698c34f7924d3bb8e2d08e64d"
server_key_version: "encryptionkey@0"
I20260928 02:38:35.550459 22249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:35.563686 22249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:35.565721 22249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:35.565851 22249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:35.569393 22249 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34961
I20260928 02:38:35.569440 22316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34961 every 8 connection(s)
I20260928 02:38:35.569734 22249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:38:35.570516 22317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:35.571664 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 22249
I20260928 02:38:35.571764 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:38:35.571915 392 external_mini_cluster.cc:1428] Setting key 8f843eb5949e09f10d8b4879fc413789
I20260928 02:38:35.572932 22317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6480d548d1d545bf8b24d7cbe0ce5eae: Bootstrap starting.
I20260928 02:38:35.573596 22317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6480d548d1d545bf8b24d7cbe0ce5eae: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:35.573912 22317 log.cc:824] T 00000000000000000000000000000000 P 6480d548d1d545bf8b24d7cbe0ce5eae: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:35.574568 22317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6480d548d1d545bf8b24d7cbe0ce5eae: No bootstrap required, opened a new log
I20260928 02:38:35.576736 22317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6480d548d1d545bf8b24d7cbe0ce5eae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6480d548d1d545bf8b24d7cbe0ce5eae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34961 } }
I20260928 02:38:35.576879 22317 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6480d548d1d545bf8b24d7cbe0ce5eae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:35.576910 22317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6480d548d1d545bf8b24d7cbe0ce5eae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6480d548d1d545bf8b24d7cbe0ce5eae, State: Initialized, Role: FOLLOWER
I20260928 02:38:35.577073 22317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6480d548d1d545bf8b24d7cbe0ce5eae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6480d548d1d545bf8b24d7cbe0ce5eae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34961 } }
I20260928 02:38:35.577137 22317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6480d548d1d545bf8b24d7cbe0ce5eae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:38:35.577158 22317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6480d548d1d545bf8b24d7cbe0ce5eae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:38:35.577203 22317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6480d548d1d545bf8b24d7cbe0ce5eae [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:35.577739 22317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6480d548d1d545bf8b24d7cbe0ce5eae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6480d548d1d545bf8b24d7cbe0ce5eae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34961 } }
I20260928 02:38:35.577826 22317 leader_election.cc:302] T 00000000000000000000000000000000 P 6480d548d1d545bf8b24d7cbe0ce5eae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6480d548d1d545bf8b24d7cbe0ce5eae; no voters:
I20260928 02:38:35.577977 22317 leader_election.cc:288] T 00000000000000000000000000000000 P 6480d548d1d545bf8b24d7cbe0ce5eae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:38:35.577994 22322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6480d548d1d545bf8b24d7cbe0ce5eae [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:35.578100 22322 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6480d548d1d545bf8b24d7cbe0ce5eae [term 1 LEADER]: Becoming Leader. State: Replica: 6480d548d1d545bf8b24d7cbe0ce5eae, State: Running, Role: LEADER
I20260928 02:38:35.578179 22322 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6480d548d1d545bf8b24d7cbe0ce5eae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6480d548d1d545bf8b24d7cbe0ce5eae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34961 } }
I20260928 02:38:35.578229 22317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6480d548d1d545bf8b24d7cbe0ce5eae [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 02:38:35 dist-test-slave-fz47 krb5kdc[22236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:38:35.578478 22322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6480d548d1d545bf8b24d7cbe0ce5eae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6480d548d1d545bf8b24d7cbe0ce5eae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6480d548d1d545bf8b24d7cbe0ce5eae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34961 } } }
I20260928 02:38:35.578536 22322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6480d548d1d545bf8b24d7cbe0ce5eae [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:35.578656 22322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6480d548d1d545bf8b24d7cbe0ce5eae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6480d548d1d545bf8b24d7cbe0ce5eae. Latest consensus state: current_term: 1 leader_uuid: "6480d548d1d545bf8b24d7cbe0ce5eae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6480d548d1d545bf8b24d7cbe0ce5eae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34961 } } }
I20260928 02:38:35.578699 22322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6480d548d1d545bf8b24d7cbe0ce5eae [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:35.578811 22326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:38:35.579485 22326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:38:35.581250 22326 catalog_manager.cc:1347] Generated new cluster ID: 869a773a1d154fdb861eecdb2dcdc9ae
I20260928 02:38:35.581322 22326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:38:35.582679 22320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:35.572389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48717 (local address 127.0.98.62:34961)
0928 02:38:35.572566 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:35.572573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:35.572596 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:38:35.572993 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:35.572999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:35.573044 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:35.573176 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:35.573976 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:35.574943 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:35.576010 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:35.576399 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:35.576489 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:35.579090 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:35.579094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:35.579095 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:35.579107 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:35.581126 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:35.581536 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:35.581539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:35.581540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:35.581643 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:35.581839 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:35.581840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:35.581841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:35.582101 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:35.582239 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:35.582363 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 02:38:35.582492 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
I20260928 02:38:35.583475 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:38:35.590816 22320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:35.585525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56109 (local address 127.0.98.62:34961)
0928 02:38:35.585557 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:35.585559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:35.585699 (+ 140us) server_negotiation.cc:407] Connection header received
0928 02:38:35.585739 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:35.585740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:35.585749 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:35.585791 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:35.586251 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:35.586731 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:35.587377 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:35.587600 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:35.587651 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:35.588655 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:35.588656 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:35.588656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:35.588662 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:35.589859 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:35.590163 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:35.590164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:35.590165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:35.590214 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:35.590377 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:35.590378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:35.590378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:35.590486 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:35.590564 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:35.590671 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 02:38:35.590735 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260928 02:38:35.596685 22326 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:38:35.597404 22326 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:38:35.610155 22326 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6480d548d1d545bf8b24d7cbe0ce5eae: Generated new TSK 0
I20260928 02:38:35.610347 22326 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:38:35.616086 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 22249
2026-09-28T02:38:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1220 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:38:35 dist-test-slave-fz47 krb5kdc[22348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:38:35 dist-test-slave-fz47 krb5kdc[22348](info): set up 2 sockets
Sep 28 02:38:35 dist-test-slave-fz47 krb5kdc[22348](info): commencing operation
krb5kdc: starting...
W20260928 02:38:36.581241 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.003s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:38:36 dist-test-slave-fz47 krb5kdc[22348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:38:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:38:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 02:38:36.691449 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32891
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:36.768673 22361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:36.768874 22361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:36.768903 22361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:36.770072 22361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:36.770118 22361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:36.770136 22361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:36.770152 22361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:36.770166 22361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:36.771615 22361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:36.771857 22361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:36.772063 22361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:36.776598 22367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:36.776585 22366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:36.776849 22361 server_base.cc:1034] running on GCE node
W20260928 02:38:36.776849 22369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:36.777112 22361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:36.777423 22361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:36.778581 22361 hybrid_clock.cc:648] HybridClock initialized: now 1790563116778559 us; error 34 us; skew 500 ppm
Sep 28 02:38:36 dist-test-slave-fz47 krb5kdc[22348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:36.786324 22361 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:38:36.786777 22361 webserver.cc:460] Webserver started at http://127.0.98.62:41007/ using document root <none> and password file <none>
I20260928 02:38:36.786971 22361 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:36.787011 22361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:36.787102 22361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:36.788111 22361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "a4d592605c044295a51163596c9d99dc"
format_stamp: "Formatted at 2026-09-28 02:38:36 on dist-test-slave-fz47"
server_key: "45268d6ce4a4ffe3ead384b2fe1cdcd9"
server_key_iv: "522eda852486294939ce82f52b0041a7"
server_key_version: "encryptionkey@0"
I20260928 02:38:36.788405 22361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "a4d592605c044295a51163596c9d99dc"
format_stamp: "Formatted at 2026-09-28 02:38:36 on dist-test-slave-fz47"
server_key: "45268d6ce4a4ffe3ead384b2fe1cdcd9"
server_key_iv: "522eda852486294939ce82f52b0041a7"
server_key_version: "encryptionkey@0"
I20260928 02:38:36.789634 22361 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 02:38:36.790416 22376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:36.790630 22361 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:38:36.790724 22361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "a4d592605c044295a51163596c9d99dc"
format_stamp: "Formatted at 2026-09-28 02:38:36 on dist-test-slave-fz47"
server_key: "45268d6ce4a4ffe3ead384b2fe1cdcd9"
server_key_iv: "522eda852486294939ce82f52b0041a7"
server_key_version: "encryptionkey@0"
I20260928 02:38:36.790784 22361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:36.799752 22361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:36.801841 22361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:36.801956 22361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:36.805591 22361 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32891
I20260928 02:38:36.805622 22428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32891 every 8 connection(s)
I20260928 02:38:36.805948 22361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:38:36.806808 22429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:36.809026 22429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4d592605c044295a51163596c9d99dc: Bootstrap starting.
I20260928 02:38:36.809633 22429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4d592605c044295a51163596c9d99dc: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:36.809887 22429 log.cc:824] T 00000000000000000000000000000000 P a4d592605c044295a51163596c9d99dc: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:36.810559 22429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4d592605c044295a51163596c9d99dc: No bootstrap required, opened a new log
I20260928 02:38:36.812108 22429 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4d592605c044295a51163596c9d99dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d592605c044295a51163596c9d99dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32891 } }
I20260928 02:38:36.812250 22429 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4d592605c044295a51163596c9d99dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:36.812280 22429 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4d592605c044295a51163596c9d99dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4d592605c044295a51163596c9d99dc, State: Initialized, Role: FOLLOWER
I20260928 02:38:36.812412 22429 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4d592605c044295a51163596c9d99dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d592605c044295a51163596c9d99dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32891 } }
I20260928 02:38:36.812484 22429 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4d592605c044295a51163596c9d99dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:38:36.812512 22429 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4d592605c044295a51163596c9d99dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:38:36.812562 22429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4d592605c044295a51163596c9d99dc [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:36.813076 22429 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4d592605c044295a51163596c9d99dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d592605c044295a51163596c9d99dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32891 } }
I20260928 02:38:36.813169 22429 leader_election.cc:302] T 00000000000000000000000000000000 P a4d592605c044295a51163596c9d99dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4d592605c044295a51163596c9d99dc; no voters:
I20260928 02:38:36.813354 22429 leader_election.cc:288] T 00000000000000000000000000000000 P a4d592605c044295a51163596c9d99dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:38:36.813400 22432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4d592605c044295a51163596c9d99dc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:36.813552 22432 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4d592605c044295a51163596c9d99dc [term 1 LEADER]: Becoming Leader. State: Replica: a4d592605c044295a51163596c9d99dc, State: Running, Role: LEADER
I20260928 02:38:36.813616 22429 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4d592605c044295a51163596c9d99dc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:38:36.813642 22432 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4d592605c044295a51163596c9d99dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d592605c044295a51163596c9d99dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32891 } }
I20260928 02:38:36.814049 22434 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d592605c044295a51163596c9d99dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4d592605c044295a51163596c9d99dc. Latest consensus state: current_term: 1 leader_uuid: "a4d592605c044295a51163596c9d99dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d592605c044295a51163596c9d99dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32891 } } }
I20260928 02:38:36.814165 22434 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d592605c044295a51163596c9d99dc [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:36.814153 22433 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d592605c044295a51163596c9d99dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4d592605c044295a51163596c9d99dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d592605c044295a51163596c9d99dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32891 } } }
I20260928 02:38:36.814262 22433 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d592605c044295a51163596c9d99dc [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:36.814499 22441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:38:36.815037 22441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:38:36.815730 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 22361
I20260928 02:38:36.815901 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:38:36.816079 392 external_mini_cluster.cc:1428] Setting key 6f0ca746ce8ed5c9c0f9ae98d436f6f3
I20260928 02:38:36.816870 22441 catalog_manager.cc:1347] Generated new cluster ID: 301fd06a540f42dea30b5c33e3606c46
I20260928 02:38:36.816913 22441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:38:36.822281 22452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:36.816570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36675 (local address 127.0.98.62:32891)
0928 02:38:36.816718 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:36.816726 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:36.817083 (+ 357us) server_negotiation.cc:407] Connection header received
0928 02:38:36.817217 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:36.817224 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:36.817300 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:36.817435 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:36.817669 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:36.819043 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:36.820178 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:36.820767 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:36.820857 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:36.821637 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:36.821641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:36.821641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:36.821837 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:36.821875 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:36.821889 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:36.821947 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:36.822031 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 02:38:36.822090 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
I20260928 02:38:36.823262 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:38:36.828120 22452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:36.824725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49043 (local address 127.0.98.62:32891)
0928 02:38:36.824786 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:36.824787 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:36.824993 (+ 206us) server_negotiation.cc:407] Connection header received
0928 02:38:36.825017 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:36.825018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:36.825027 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:36.825091 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:36.825190 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:36.825953 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:36.826866 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:36.827187 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:36.827255 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:36.827879 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:36.827880 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:36.827881 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:36.827885 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:36.827899 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:36.827904 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:36.827959 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:36.828017 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 02:38:36.828041 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 02:38:36.829965 22441 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:38:36.830397 22441 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:38:36.834503 22441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4d592605c044295a51163596c9d99dc: Generated new TSK 0
I20260928 02:38:36.834640 22441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:38:36.839540 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 22361
2026-09-28T02:38:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1223 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:38:36 dist-test-slave-fz47 krb5kdc[22460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:38:36 dist-test-slave-fz47 krb5kdc[22460](info): set up 2 sockets
Sep 28 02:38:36 dist-test-slave-fz47 krb5kdc[22460](info): commencing operation
krb5kdc: starting...
W20260928 02:38:37.781040 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:38:37 dist-test-slave-fz47 krb5kdc[22460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:38:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:38:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 02:38:37.889719 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40675
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:37.967234 22473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:37.967422 22473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:37.967442 22473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:37.968509 22473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:37.968544 22473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:37.968556 22473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:37.968566 22473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:37.968577 22473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:37.969964 22473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:37.970130 22473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:37.970305 22473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:37.974793 22481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:37.974897 22473 server_base.cc:1034] running on GCE node
W20260928 02:38:37.974793 22478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:37.974902 22479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:37.975248 22473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:37.975490 22473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:37.976629 22473 hybrid_clock.cc:648] HybridClock initialized: now 1790563117976617 us; error 28 us; skew 500 ppm
Sep 28 02:38:37 dist-test-slave-fz47 krb5kdc[22460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:37.978889 22473 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:38:37.979292 22473 webserver.cc:460] Webserver started at http://127.0.98.62:35687/ using document root <none> and password file <none>
I20260928 02:38:37.979491 22473 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:37.979533 22473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:37.979642 22473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:37.980614 22473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "4acb5a5b276e4698a65ce63c1df62459"
format_stamp: "Formatted at 2026-09-28 02:38:37 on dist-test-slave-fz47"
server_key: "cef5fda6cda2144cbfa9eadf2b3e60f8"
server_key_iv: "378e44311612357254391ed0d927ef9f"
server_key_version: "encryptionkey@0"
I20260928 02:38:37.980911 22473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "4acb5a5b276e4698a65ce63c1df62459"
format_stamp: "Formatted at 2026-09-28 02:38:37 on dist-test-slave-fz47"
server_key: "cef5fda6cda2144cbfa9eadf2b3e60f8"
server_key_iv: "378e44311612357254391ed0d927ef9f"
server_key_version: "encryptionkey@0"
I20260928 02:38:37.982152 22473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:38:37.982802 22488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:37.982970 22473 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:37.983039 22473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "4acb5a5b276e4698a65ce63c1df62459"
format_stamp: "Formatted at 2026-09-28 02:38:37 on dist-test-slave-fz47"
server_key: "cef5fda6cda2144cbfa9eadf2b3e60f8"
server_key_iv: "378e44311612357254391ed0d927ef9f"
server_key_version: "encryptionkey@0"
I20260928 02:38:37.983103 22473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:37.986225 22473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:37.988101 22473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:37.988210 22473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:37.991791 22473 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40675
I20260928 02:38:37.991850 22540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40675 every 8 connection(s)
I20260928 02:38:37.992116 22473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:38:37.992975 22541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:37.993497 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 22473
I20260928 02:38:37.993603 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:38:37.993777 392 external_mini_cluster.cc:1428] Setting key e4dfd78ce7883e669583c0f501144ad2
I20260928 02:38:37.995728 22541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4acb5a5b276e4698a65ce63c1df62459: Bootstrap starting.
I20260928 02:38:37.995986 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:37.994277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52985 (local address 127.0.98.62:40675)
0928 02:38:37.994440 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:37.994447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:37.994471 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:38:37.994837 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:37.994842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:37.994865 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:37.994979 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:37.995566 (+ 587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:37.995569 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:37.995569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:37.995573 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:37.995611 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:37.995621 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:37.995672 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:37.995770 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 02:38:37.995800 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
I20260928 02:38:37.996402 22541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4acb5a5b276e4698a65ce63c1df62459: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:37.996688 22541 log.cc:824] T 00000000000000000000000000000000 P 4acb5a5b276e4698a65ce63c1df62459: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:37.996866 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:38:37.997439 22541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4acb5a5b276e4698a65ce63c1df62459: No bootstrap required, opened a new log
I20260928 02:38:37.999193 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:37.998283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47263 (local address 127.0.98.62:40675)
0928 02:38:37.998322 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:37.998324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:37.998430 (+ 106us) server_negotiation.cc:407] Connection header received
0928 02:38:37.998472 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:37.998473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:37.998478 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:37.998527 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:37.998997 (+ 470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:37.998998 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:37.998998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:38:37.999002 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:37.999016 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:38:37.999020 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:37.999086 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:37.999119 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 02:38:37.999131 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 02:38:37.999581 22541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4acb5a5b276e4698a65ce63c1df62459 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4acb5a5b276e4698a65ce63c1df62459" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40675 } }
I20260928 02:38:37.999703 22541 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4acb5a5b276e4698a65ce63c1df62459 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:38:37.999732 22541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4acb5a5b276e4698a65ce63c1df62459 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acb5a5b276e4698a65ce63c1df62459, State: Initialized, Role: FOLLOWER
I20260928 02:38:37.999841 22541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4acb5a5b276e4698a65ce63c1df62459 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4acb5a5b276e4698a65ce63c1df62459" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40675 } }
I20260928 02:38:37.999918 22541 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4acb5a5b276e4698a65ce63c1df62459 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:38:37.999953 22541 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4acb5a5b276e4698a65ce63c1df62459 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:38:37.999997 22541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4acb5a5b276e4698a65ce63c1df62459 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:38.000470 22541 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4acb5a5b276e4698a65ce63c1df62459 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4acb5a5b276e4698a65ce63c1df62459" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40675 } }
I20260928 02:38:38.000552 22541 leader_election.cc:302] T 00000000000000000000000000000000 P 4acb5a5b276e4698a65ce63c1df62459 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4acb5a5b276e4698a65ce63c1df62459; no voters:
I20260928 02:38:38.000698 22541 leader_election.cc:288] T 00000000000000000000000000000000 P 4acb5a5b276e4698a65ce63c1df62459 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:38:38.000779 22551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4acb5a5b276e4698a65ce63c1df62459 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:38.000905 22551 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4acb5a5b276e4698a65ce63c1df62459 [term 1 LEADER]: Becoming Leader. State: Replica: 4acb5a5b276e4698a65ce63c1df62459, State: Running, Role: LEADER
I20260928 02:38:38.000967 22541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4acb5a5b276e4698a65ce63c1df62459 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:38:38.001031 22551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4acb5a5b276e4698a65ce63c1df62459 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4acb5a5b276e4698a65ce63c1df62459" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40675 } }
I20260928 02:38:38.001442 22553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4acb5a5b276e4698a65ce63c1df62459 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4acb5a5b276e4698a65ce63c1df62459. Latest consensus state: current_term: 1 leader_uuid: "4acb5a5b276e4698a65ce63c1df62459" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4acb5a5b276e4698a65ce63c1df62459" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40675 } } }
I20260928 02:38:38.001533 22553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4acb5a5b276e4698a65ce63c1df62459 [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:38.001427 22552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4acb5a5b276e4698a65ce63c1df62459 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4acb5a5b276e4698a65ce63c1df62459" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4acb5a5b276e4698a65ce63c1df62459" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40675 } } }
I20260928 02:38:38.001611 22552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4acb5a5b276e4698a65ce63c1df62459 [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:38.001909 22559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:38:38.002347 22559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:38:38.003796 22559 catalog_manager.cc:1347] Generated new cluster ID: da35d6b3d02c4244934596ac26cc673c
I20260928 02:38:38.003847 22559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:38:38.021286 22559 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:38:38.021441 22559 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:38:38.039402 22559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4acb5a5b276e4698a65ce63c1df62459: Generated new TSK 0
I20260928 02:38:38.039590 22559 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:38:38.050673 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 22473
2026-09-28T02:38:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1211 ms)
[----------] 18 tests from AuthTokenIssuingTest (21671 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:38:38 dist-test-slave-fz47 krb5kdc[22572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:38:38 dist-test-slave-fz47 krb5kdc[22572](info): set up 2 sockets
Sep 28 02:38:38 dist-test-slave-fz47 krb5kdc[22572](info): commencing operation
krb5kdc: starting...
W20260928 02:38:38.989089 392 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:38:39 dist-test-slave-fz47 krb5kdc[22572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:38:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:38:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 02:38:39.097733 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39673
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39673,127.0.98.61:46615,127.0.98.60:34487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:39.175429 22585 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:38:39.175625 22585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:39.175640 22585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:39.175658 22585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:39.176808 22585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:39.176847 22585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:39.176859 22585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:39.176870 22585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:39.176882 22585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:39.178298 22585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39673,127.0.98.61:46615,127.0.98.60:34487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:39.178481 22585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:39.178647 22585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:39.183190 22593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:39.183316 22591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:39.183167 22585 server_base.cc:1034] running on GCE node
W20260928 02:38:39.183162 22590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:39.183594 22585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:39.183866 22585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:39.185007 22585 hybrid_clock.cc:648] HybridClock initialized: now 1790563119184998 us; error 26 us; skew 500 ppm
Sep 28 02:38:39 dist-test-slave-fz47 krb5kdc[22572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:39.187331 22585 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:38:39.187810 22585 webserver.cc:460] Webserver started at http://127.0.98.62:33887/ using document root <none> and password file <none>
I20260928 02:38:39.188014 22585 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:39.188059 22585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:39.188184 22585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:39.189153 22585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "4e39df79b28b44049c020a7011c96682"
format_stamp: "Formatted at 2026-09-28 02:38:39 on dist-test-slave-fz47"
server_key: "1517b7d2773d3ce616a8ec8f2096f7b2"
server_key_iv: "4d6773c08968994d3aaa3c1535519aed"
server_key_version: "encryptionkey@0"
I20260928 02:38:39.189503 22585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "4e39df79b28b44049c020a7011c96682"
format_stamp: "Formatted at 2026-09-28 02:38:39 on dist-test-slave-fz47"
server_key: "1517b7d2773d3ce616a8ec8f2096f7b2"
server_key_iv: "4d6773c08968994d3aaa3c1535519aed"
server_key_version: "encryptionkey@0"
I20260928 02:38:39.190742 22585 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:38:39.191483 22600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:39.191638 22585 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:39.191720 22585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "4e39df79b28b44049c020a7011c96682"
format_stamp: "Formatted at 2026-09-28 02:38:39 on dist-test-slave-fz47"
server_key: "1517b7d2773d3ce616a8ec8f2096f7b2"
server_key_iv: "4d6773c08968994d3aaa3c1535519aed"
server_key_version: "encryptionkey@0"
I20260928 02:38:39.191807 22585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:39.210861 22585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:39.212623 22585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:39.212765 22585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:39.216454 22585 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39673
I20260928 02:38:39.216501 22652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39673 every 8 connection(s)
I20260928 02:38:39.216807 22585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:38:39.217697 22653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:39.219137 22653 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39673 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:38:39.221796 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 22585
I20260928 02:38:39.221879 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:38:39.222038 392 external_mini_cluster.cc:1428] Setting key 3f3d9df85d1716cc3c82c6a50abcdd98
Sep 28 02:38:39 dist-test-slave-fz47 krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:38:39.228297 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.219786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58108 (local address 127.0.98.62:39673)
0928 02:38:39.219908 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:39.219917 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:39.220015 (+ 98us) server_negotiation.cc:407] Connection header received
0928 02:38:39.220548 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:39.220550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:39.220566 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:39.220637 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:39.221153 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.222069 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.222902 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.223198 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.223302 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:39.225422 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:39.225429 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:39.225429 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:39.225435 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:39.226845 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:39.227281 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:39.227284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:39.227284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:39.227354 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:39.227581 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:39.227581 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:39.227582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:39.227761 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:39.227837 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:39.228049 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 02:38:39.228143 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":60,"threads_started":1}
I20260928 02:38:39.228372 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.219714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39673 (local address 127.0.0.1:58108)
0928 02:38:39.219883 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:39.219925 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:39.220398 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:39.220660 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:39.220663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:39.220919 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:39.221059 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:39.221063 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.222158 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.222159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:39.222818 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:39.222820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.223290 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.223291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:39.223355 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:39.223799 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:39.223807 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:39.225337 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:39.226933 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:39.226940 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:39.226942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:39.227209 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:39.227416 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:39.227417 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:39.227417 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:39.227520 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:39.227847 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:39.227854 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:39.227926 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 02:38:39.228024 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 02:38:39.228236 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":61,"thread_start_us":75,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260928 02:38:39.229053 22653 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46615 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:38:39.229350 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.229239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46615 (local address 127.0.0.1:50346)
0928 02:38:39.229289 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:39.229333 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 02:38:39.229476 22604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46615: connect: Connection refused (error 111)
I20260928 02:38:39.229880 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.229796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46615 (local address 127.0.0.1:50348)
0928 02:38:39.229834 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:39.229859 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 02:38:39.229983 22653 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46615: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46615: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260928 02:38:39.244174 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46615
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39673,127.0.98.61:46615,127.0.98.60:34487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:38:39.279198 22653 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46615 } attempt: 1
I20260928 02:38:39.279695 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.279526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46615 (local address 127.0.0.1:50362)
0928 02:38:39.279597 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:39.279652 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 02:38:39.280061 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.279971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46615 (local address 127.0.0.1:50368)
0928 02:38:39.280015 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:39.280041 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 02:38:39.280169 22653 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46615: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46615: connect: Connection refused (error 111)
W20260928 02:38:39.322016 22660 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:38:39.322206 22660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:39.322220 22660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:39.322238 22660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:39.323334 22660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:39.323371 22660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:39.323385 22660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:39.323395 22660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:39.323405 22660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:39.324841 22660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39673,127.0.98.61:46615,127.0.98.60:34487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:39.325050 22660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:39.325248 22660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:39.329806 22666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:39.329842 22668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:39.329751 22665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:39.329941 22660 server_base.cc:1034] running on GCE node
I20260928 02:38:39.330147 22660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:39.330415 22660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:39.331554 22660 hybrid_clock.cc:648] HybridClock initialized: now 1790563119331545 us; error 30 us; skew 500 ppm
Sep 28 02:38:39 dist-test-slave-fz47 krb5kdc[22572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:39.333819 22660 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260928 02:38:39.334241 22660 webserver.cc:460] Webserver started at http://127.0.98.61:37285/ using document root <none> and password file <none>
I20260928 02:38:39.334430 22660 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:39.334476 22660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:39.334596 22660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:39.335556 22660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-1/data/instance:
uuid: "932214c249b94adcbf2480b93ec96202"
format_stamp: "Formatted at 2026-09-28 02:38:39 on dist-test-slave-fz47"
server_key: "6299aa4da870189ccc69dbd1db3979e1"
server_key_iv: "8193239e9d1a8bfa137560f0b1471f58"
server_key_version: "encryptionkey@0"
I20260928 02:38:39.335894 22660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-1/wal/instance:
uuid: "932214c249b94adcbf2480b93ec96202"
format_stamp: "Formatted at 2026-09-28 02:38:39 on dist-test-slave-fz47"
server_key: "6299aa4da870189ccc69dbd1db3979e1"
server_key_iv: "8193239e9d1a8bfa137560f0b1471f58"
server_key_version: "encryptionkey@0"
I20260928 02:38:39.337037 22660 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 02:38:39.337726 22675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:39.337878 22660 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:39.337955 22660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-1/wal
uuid: "932214c249b94adcbf2480b93ec96202"
format_stamp: "Formatted at 2026-09-28 02:38:39 on dist-test-slave-fz47"
server_key: "6299aa4da870189ccc69dbd1db3979e1"
server_key_iv: "8193239e9d1a8bfa137560f0b1471f58"
server_key_version: "encryptionkey@0"
I20260928 02:38:39.338021 22660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:39.348301 22653 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46615 } attempt: 2
I20260928 02:38:39.348681 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.348532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46615 (local address 127.0.0.1:50384)
0928 02:38:39.348599 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:39.348638 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 02:38:39.348896 22660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:39.349031 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.348933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46615 (local address 127.0.0.1:50386)
0928 02:38:39.348980 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:39.349008 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 02:38:39.349355 22653 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46615: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46615: connect: Connection refused (error 111)
I20260928 02:38:39.350594 22660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:39.350694 22660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:39.354298 22660 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46615
I20260928 02:38:39.354328 22727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46615 every 8 connection(s)
I20260928 02:38:39.354614 22660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-1/data/info.pb
I20260928 02:38:39.355465 22728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:39.356863 22728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39673 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:38:39.358258 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 22660
I20260928 02:38:39.358361 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-1/wal/instance
I20260928 02:38:39.358510 392 external_mini_cluster.cc:1428] Setting key 48b38067825a32b6e643f1fbf11353cb
Sep 28 02:38:39 dist-test-slave-fz47 krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260928 02:38:39.365128 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.357495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58120 (local address 127.0.98.62:39673)
0928 02:38:39.357545 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:39.357547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:39.357736 (+ 189us) server_negotiation.cc:407] Connection header received
0928 02:38:39.358228 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:39.358229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:39.358243 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:39.358292 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:39.358951 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.359425 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.360352 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.360555 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.360609 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:39.362630 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:39.362631 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:39.362632 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:39.362639 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:39.363896 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:39.364259 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:39.364260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:39.364260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:39.364313 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:39.364528 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:39.364529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:39.364529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:39.364636 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:39.364692 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:39.364950 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 02:38:39.365040 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 02:38:39.365281 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.357460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39673 (local address 127.0.0.1:58120)
0928 02:38:39.357625 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:39.357658 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:39.358131 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:39.358369 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:39.358374 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:39.358669 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:39.358862 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:39.358866 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.359509 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.359510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:39.360258 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:39.360260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.360648 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.360649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:39.360763 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:39.361216 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:39.361224 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:39.362541 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:39.363973 (+ 1432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:39.363980 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:39.363982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:39.364187 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:39.364372 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:39.364373 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:39.364373 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:39.364470 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:39.364695 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:39.364702 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:39.364831 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 02:38:39.364916 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 02:38:39.365055 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":75,"threads_started":1}
I20260928 02:38:39.365859 22728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46615 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 28 02:38:39 dist-test-slave-fz47 krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 02:38:39.372779 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.366070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46615 (local address 127.0.0.1:50390)
0928 02:38:39.366115 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:39.366125 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:39.366162 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:39.366386 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:39.366387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:39.366506 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:39.366554 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:39.366555 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.367433 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.367435 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:39.368135 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:39.368138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.368615 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.368617 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:39.368698 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:39.369049 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:39.369053 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:39.370202 (+ 1149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:39.371795 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:39.371796 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:39.371796 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:39.371997 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:39.372179 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:39.372179 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:39.372180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:39.372241 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:39.372482 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:39.372483 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:39.372526 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 02:38:39.372595 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 02:38:39.372685 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 02:38:39.372853 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.366141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50390 (local address 127.0.98.61:46615)
0928 02:38:39.366259 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:39.366263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:39.366283 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:38:39.366296 (+ 13us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:39.366298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:39.366307 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:39.366354 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:39.366618 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.367352 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.368220 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.368546 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.368595 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:39.370287 (+ 1692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:39.370294 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:39.370294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:39.370300 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:39.371706 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:39.372068 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:39.372070 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:39.372070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:39.372116 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:39.372304 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:39.372305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:39.372305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:39.372419 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:39.372473 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:39.372620 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 02:38:39.372746 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":99,"thread_start_us":74,"threads_started":1}
I20260928 02:38:39.373426 22728 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34487 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:38:39.373745 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.373623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34487 (local address 127.0.0.1:53962)
0928 02:38:39.373673 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:39.373721 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 02:38:39.373879 22679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34487: connect: Connection refused (error 111)
I20260928 02:38:39.374295 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.374209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34487 (local address 127.0.0.1:53966)
0928 02:38:39.374244 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:39.374280 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 02:38:39.374387 22728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34487: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34487: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260928 02:38:39.382026 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34487
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39673,127.0.98.61:46615,127.0.98.60:34487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:38:39.423605 22728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34487 } attempt: 1
I20260928 02:38:39.424096 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.423957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34487 (local address 127.0.0.1:53978)
0928 02:38:39.424017 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:39.424062 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 02:38:39.424454 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.424353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34487 (local address 127.0.0.1:53988)
0928 02:38:39.424401 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:39.424429 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 02:38:39.424556 22728 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34487: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34487: connect: Connection refused (error 111)
I20260928 02:38:39.440495 22653 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46615 } attempt: 3
Sep 28 02:38:39 dist-test-slave-fz47 krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 02:38:39.448117 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.441034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46615 (local address 127.0.0.1:50400)
0928 02:38:39.441096 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:39.441108 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:39.441153 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:39.441431 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:39.441433 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:39.441630 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:39.441718 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:39.441721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.442501 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.442503 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:39.443301 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:39.443305 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.443667 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.443668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:39.443746 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:39.444162 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:39.444168 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:39.445521 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:39.447161 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:39.447162 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:39.447163 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:39.447331 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:39.447551 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:39.447553 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:39.447553 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:39.447616 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:39.447886 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:39.447887 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:39.447929 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 02:38:39.447975 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 02:38:39.448050 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 02:38:39.448172 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.441081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50400 (local address 127.0.98.61:46615)
0928 02:38:39.441230 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:39.441232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:39.441239 (+ 7us) server_negotiation.cc:407] Connection header received
0928 02:38:39.441287 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:39.441288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:39.441301 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:39.441373 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:39.441819 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.442395 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.443395 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.443599 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.443651 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:39.445606 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:39.445608 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:39.445609 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:39.445618 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:39.447073 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:39.447401 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:39.447402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:39.447403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:39.447463 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:39.447688 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:39.447689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:39.447690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:39.447822 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:39.447894 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:39.448003 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 02:38:39.448082 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260928 02:38:39.448536 22653 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34487 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:38:39.448858 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.448764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34487 (local address 127.0.0.1:53998)
0928 02:38:39.448812 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:39.448837 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 02:38:39.449162 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.449086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34487 (local address 127.0.0.1:54010)
0928 02:38:39.449125 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:39.449147 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 02:38:39.449246 22653 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34487: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34487: connect: Connection refused (error 111)
W20260928 02:38:39.462371 22735 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:38:39.462559 22735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:39.462574 22735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:39.462591 22735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:39.463670 22735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:39.463706 22735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:39.463719 22735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:39.463730 22735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:39.463749 22735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:39.465135 22735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39673,127.0.98.61:46615,127.0.98.60:34487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:39.465348 22735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:39.465552 22735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:39.469969 22740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:39.470019 22743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:39.469969 22741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:39.470161 22735 server_base.cc:1034] running on GCE node
I20260928 02:38:39.470352 22735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:39.470597 22735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:39.471737 22735 hybrid_clock.cc:648] HybridClock initialized: now 1790563119471721 us; error 29 us; skew 500 ppm
Sep 28 02:38:39 dist-test-slave-fz47 krb5kdc[22572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:39.474010 22735 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260928 02:38:39.474436 22735 webserver.cc:460] Webserver started at http://127.0.98.60:39671/ using document root <none> and password file <none>
I20260928 02:38:39.474629 22735 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:39.474675 22735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:39.474800 22735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:39.475831 22735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-2/data/instance:
uuid: "16bf871a38c64ef191a63926ca7c697c"
format_stamp: "Formatted at 2026-09-28 02:38:39 on dist-test-slave-fz47"
server_key: "c6faf008faf818d00912a732269bbfb1"
server_key_iv: "498917c5f3fc5d870e2f31943ca58191"
server_key_version: "encryptionkey@0"
I20260928 02:38:39.476138 22735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-2/wal/instance:
uuid: "16bf871a38c64ef191a63926ca7c697c"
format_stamp: "Formatted at 2026-09-28 02:38:39 on dist-test-slave-fz47"
server_key: "c6faf008faf818d00912a732269bbfb1"
server_key_iv: "498917c5f3fc5d870e2f31943ca58191"
server_key_version: "encryptionkey@0"
I20260928 02:38:39.477330 22735 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:38:39.477972 22750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:39.478124 22735 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:39.478201 22735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-2/wal
uuid: "16bf871a38c64ef191a63926ca7c697c"
format_stamp: "Formatted at 2026-09-28 02:38:39 on dist-test-slave-fz47"
server_key: "c6faf008faf818d00912a732269bbfb1"
server_key_iv: "498917c5f3fc5d870e2f31943ca58191"
server_key_version: "encryptionkey@0"
I20260928 02:38:39.478263 22735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:39.480391 22653 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34487 } attempt: 1
I20260928 02:38:39.480779 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.480637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34487 (local address 127.0.0.1:54012)
0928 02:38:39.480689 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:39.480723 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 02:38:39.481112 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.481018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34487 (local address 127.0.0.1:54026)
0928 02:38:39.481063 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:39.481087 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 02:38:39.481189 22653 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34487: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34487: connect: Connection refused (error 111)
I20260928 02:38:39.487133 22735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:39.488648 22735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:39.488776 22735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:39.492364 22735 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34487
I20260928 02:38:39.492396 22802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34487 every 8 connection(s)
I20260928 02:38:39.492695 22735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-2/data/info.pb
I20260928 02:38:39.492717 22728 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34487 } attempt: 2
I20260928 02:38:39.493734 22803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:39.495239 22803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39673 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:38:39.495898 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 22735
I20260928 02:38:39.495999 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790562862804661-392-0/minicluster-data/master-2/wal/instance
I20260928 02:38:39.496176 392 external_mini_cluster.cc:1428] Setting key ecd0da22d0d232fa23388d180cb1959b
Sep 28 02:38:39 dist-test-slave-fz47 krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 02:38:39 dist-test-slave-fz47 krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 28 02:38:39 dist-test-slave-fz47 krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 02:38:39.502005 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.492977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34487 (local address 127.0.0.1:54030)
0928 02:38:39.493029 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:39.493039 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:39.493079 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:39.494084 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:39.494085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:39.494242 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:39.494291 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:39.494294 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.495366 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.495368 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:39.496136 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:39.496139 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.496873 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.496875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:39.496934 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:39.497405 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:39.497410 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:39.498582 (+ 1172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:39.500732 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:39.500734 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:39.500734 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:39.500913 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:39.501155 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:39.501156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:39.501157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:39.501220 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:39.501695 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:39.501696 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:39.501745 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 02:38:39.501825 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:38:39.501914 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 02:38:39.502136 22805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.493273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54030 (local address 127.0.98.60:34487)
0928 02:38:39.493451 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:39.493459 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:39.493482 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:38:39.493895 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:39.493902 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:39.493949 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:39.494085 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:39.494374 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.495270 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.496220 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.496815 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.496905 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:39.498683 (+ 1778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:39.498687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:39.498687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:39.498700 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:39.500647 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:39.500984 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:39.500987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:39.500988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:39.501077 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:39.501347 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:39.501348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:39.501348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:39.501604 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:39.501705 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:39.501883 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 02:38:39.501971 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":99,"threads_started":1}
I20260928 02:38:39.503895 22728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 932214c249b94adcbf2480b93ec96202: Bootstrap starting.
I20260928 02:38:39.504798 22728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 932214c249b94adcbf2480b93ec96202: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:39.505034 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.496494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54034 (local address 127.0.98.60:34487)
0928 02:38:39.496615 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:39.496616 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:39.496737 (+ 121us) server_negotiation.cc:407] Connection header received
0928 02:38:39.496805 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:39.496805 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:39.496815 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:39.496870 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:39.497612 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.498356 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.499248 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.499541 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.499601 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:39.501976 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:39.501977 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:39.501977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:39.501983 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:39.504050 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:39.504356 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:39.504358 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:39.504358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:39.504420 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:39.504585 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:39.504586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:39.504586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:39.504695 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:39.504834 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:39.504874 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 02:38:39.504943 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
I20260928 02:38:39.505090 22728 log.cc:824] T 00000000000000000000000000000000 P 932214c249b94adcbf2480b93ec96202: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:39.505872 22728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 932214c249b94adcbf2480b93ec96202: No bootstrap required, opened a new log
I20260928 02:38:39.506510 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.495867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58122 (local address 127.0.98.62:39673)
0928 02:38:39.496004 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:39.496006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:39.496112 (+ 106us) server_negotiation.cc:407] Connection header received
0928 02:38:39.496188 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:39.496189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:39.496200 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:39.496241 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:39.496780 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.497439 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.498475 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.498698 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.498755 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:39.500787 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:39.500788 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:39.500789 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:39.500795 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:39.502698 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:39.505655 (+ 2957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:39.505656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:39.505656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:39.505715 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:39.505952 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:39.505954 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:39.505954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:39.506086 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:39.506156 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:39.506353 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 02:38:39.506425 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260928 02:38:39.506536 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.495824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39673 (local address 127.0.0.1:58122)
0928 02:38:39.496014 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:39.496057 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:39.496128 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:39.496312 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:39.496315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:39.496618 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:39.496688 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:39.496690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.497522 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.497522 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:39.498414 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:39.498416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.498805 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.498806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:39.498891 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:39.499413 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:39.499417 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:39.500703 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:39.505303 (+ 4600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:39.505308 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:39.505310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:39.505558 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:39.505813 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:39.505814 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:39.505814 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:39.505876 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:39.506168 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:39.506174 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:39.506229 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 02:38:39.506314 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 02:38:39.506410 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":19,"thread_start_us":77,"threads_started":1}
I20260928 02:38:39.506971 22803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46615 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:38:39.507817 22728 raft_consensus.cc:348] T 00000000000000000000000000000000 P 932214c249b94adcbf2480b93ec96202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39df79b28b44049c020a7011c96682" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39673 } } peers { permanent_uuid: "932214c249b94adcbf2480b93ec96202" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46615 } } peers { permanent_uuid: "16bf871a38c64ef191a63926ca7c697c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34487 } }
I20260928 02:38:39.507948 22728 raft_consensus.cc:729] T 00000000000000000000000000000000 P 932214c249b94adcbf2480b93ec96202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 932214c249b94adcbf2480b93ec96202, State: Initialized, Role: FOLLOWER
I20260928 02:38:39.508020 22728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 932214c249b94adcbf2480b93ec96202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39df79b28b44049c020a7011c96682" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39673 } } peers { permanent_uuid: "932214c249b94adcbf2480b93ec96202" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46615 } } peers { permanent_uuid: "16bf871a38c64ef191a63926ca7c697c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34487 } }
I20260928 02:38:39.508308 22728 sys_catalog.cc:564] T 00000000000000000000000000000000 P 932214c249b94adcbf2480b93ec96202 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:38:39.508275 22811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932214c249b94adcbf2480b93ec96202 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39df79b28b44049c020a7011c96682" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39673 } } peers { permanent_uuid: "932214c249b94adcbf2480b93ec96202" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46615 } } peers { permanent_uuid: "16bf871a38c64ef191a63926ca7c697c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34487 } } }
I20260928 02:38:39.508457 22811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932214c249b94adcbf2480b93ec96202 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 02:38:39.509982 22822 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 932214c249b94adcbf2480b93ec96202: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:38:39.510077 22822 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 28 02:38:39 dist-test-slave-fz47 krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 28 02:38:39 dist-test-slave-fz47 krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 02:38:39.510803 22660 master_runner.cc:186] Error getting master registration for 127.0.98.62:39673: OK, instance_id { permanent_uuid: "4e39df79b28b44049c020a7011c96682" instance_seqno: 1790563119215378 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:38:39.512771 22660 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:38:39.513902 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.507231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46615 (local address 127.0.0.1:50404)
0928 02:38:39.507332 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:39.507342 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:39.507388 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:39.507496 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:39.507496 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:39.507626 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:39.507671 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:39.507672 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.508490 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.508491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:39.509215 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:39.509218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.509739 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.509740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:39.510016 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:39.510453 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:39.510457 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:39.511441 (+ 984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:39.512995 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:39.512996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:39.512996 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:39.513145 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:39.513340 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:39.513341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:39.513341 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:39.513393 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:39.513630 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:39.513631 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:39.513669 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:38:39.513730 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 02:38:39.513818 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 02:38:39.513933 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.507293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50404 (local address 127.0.98.61:46615)
0928 02:38:39.507334 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:39.507336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:39.507377 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:38:39.507430 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:39.507431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:39.507439 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:39.507487 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:39.507736 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.508392 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.509353 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.509613 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.509646 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:39.511526 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:39.511527 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:39.511528 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:39.511534 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:39.512914 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:39.513217 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:39.513218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:39.513218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:39.513294 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:39.513449 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:39.513450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:39.513450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:39.513566 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:39.513620 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:39.513769 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 02:38:39.513861 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:38:39.514266 22803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34487 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:38:39.515079 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.505648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58128 (local address 127.0.98.62:39673)
0928 02:38:39.505793 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:39.505795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:39.505801 (+ 6us) server_negotiation.cc:407] Connection header received
0928 02:38:39.505816 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:39.505817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:39.505826 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:39.506430 (+ 604us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:39.506483 (+ 53us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.507781 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.508628 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.508915 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.508978 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:39.511542 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:39.511544 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:39.511544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:39.511550 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:39.513710 (+ 2160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:39.514058 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:39.514059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:39.514059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:39.514120 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:39.514322 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:39.514323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:39.514323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:39.514445 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:39.514581 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:39.514917 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 02:38:39.514989 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
Sep 28 02:38:39 dist-test-slave-fz47 krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 02:38:39 dist-test-slave-fz47 krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 02:38:39.521315 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.514456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34487 (local address 127.0.0.1:54036)
0928 02:38:39.514513 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:39.514523 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:39.514558 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:39.514667 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:39.514668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:39.514794 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:39.514824 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:39.514825 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.515902 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.515903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:39.516669 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:39.516671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.517145 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.517146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:39.517200 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:39.517904 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:39.517909 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:39.519150 (+ 1241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:39.520395 (+ 1245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:39.520396 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:39.520396 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:39.520556 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:39.520731 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:39.520732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:39.520732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:39.520783 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:39.521039 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:39.521039 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:39.521073 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 02:38:39.521113 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 02:38:39.521196 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 02:38:39.521600 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.514484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54036 (local address 127.0.98.60:34487)
0928 02:38:39.514519 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:39.514520 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:39.514549 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:38:39.514600 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:39.514601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:39.514610 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:39.514666 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:39.514897 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.515816 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.516753 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.517063 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.517133 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:39.519228 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:39.519229 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:39.519230 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:39.519235 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:39.520336 (+ 1101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:39.520628 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:39.520629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:39.520629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:39.520675 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:39.520838 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:39.520839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:39.520839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:39.520968 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:39.521414 (+ 446us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:39.521454 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 02:38:39.521519 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 02:38:39.523037 22803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16bf871a38c64ef191a63926ca7c697c: Bootstrap starting.
I20260928 02:38:39.523602 22803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16bf871a38c64ef191a63926ca7c697c: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:39.523717 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.515562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50408 (local address 127.0.98.61:46615)
0928 02:38:39.515591 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:39.515592 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:39.515598 (+ 6us) server_negotiation.cc:407] Connection header received
0928 02:38:39.515614 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:39.515614 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:39.515621 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:39.515688 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:39.516174 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.516831 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.517589 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.517792 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.517844 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:39.520785 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:39.520786 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:39.520787 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:39.520794 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:39.522313 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:39.522672 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:39.522673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:39.522673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:39.522724 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:39.522928 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:39.522929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:39.522930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:39.523047 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:39.523169 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:39.523556 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 02:38:39.523629 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 02:38:39.523898 22803 log.cc:824] T 00000000000000000000000000000000 P 16bf871a38c64ef191a63926ca7c697c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:39.524642 22803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16bf871a38c64ef191a63926ca7c697c: No bootstrap required, opened a new log
I20260928 02:38:39.525303 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:38:39.526207 22803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16bf871a38c64ef191a63926ca7c697c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39df79b28b44049c020a7011c96682" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39673 } } peers { permanent_uuid: "932214c249b94adcbf2480b93ec96202" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46615 } } peers { permanent_uuid: "16bf871a38c64ef191a63926ca7c697c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34487 } }
I20260928 02:38:39.526352 22803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16bf871a38c64ef191a63926ca7c697c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16bf871a38c64ef191a63926ca7c697c, State: Initialized, Role: FOLLOWER
I20260928 02:38:39.526436 22803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16bf871a38c64ef191a63926ca7c697c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39df79b28b44049c020a7011c96682" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39673 } } peers { permanent_uuid: "932214c249b94adcbf2480b93ec96202" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46615 } } peers { permanent_uuid: "16bf871a38c64ef191a63926ca7c697c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34487 } }
I20260928 02:38:39.526726 22803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16bf871a38c64ef191a63926ca7c697c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:38:39.526716 22825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16bf871a38c64ef191a63926ca7c697c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39df79b28b44049c020a7011c96682" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39673 } } peers { permanent_uuid: "932214c249b94adcbf2480b93ec96202" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46615 } } peers { permanent_uuid: "16bf871a38c64ef191a63926ca7c697c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34487 } } }
I20260928 02:38:39.526914 22825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16bf871a38c64ef191a63926ca7c697c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:38:39.526250 22627 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4e39df79b28b44049c020a7011c96682"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58128
W20260928 02:38:39.528718 22836 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 16bf871a38c64ef191a63926ca7c697c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:38:39.528828 22836 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:38:39.528571 22735 master_runner.cc:186] Error getting master registration for 127.0.98.62:39673: OK, instance_id { permanent_uuid: "4e39df79b28b44049c020a7011c96682" instance_seqno: 1790563119215378 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:38:39.530682 22735 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:38:39.556344 22653 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34487 } attempt: 2
Sep 28 02:38:39 dist-test-slave-fz47 krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 02:38:39.564316 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.556689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34487 (local address 127.0.0.1:54050)
0928 02:38:39.556752 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:39.556763 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:39.556804 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:39.556968 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:39.556969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:39.557173 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:39.557226 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:39.557228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.558196 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.558197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:39.559011 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:39.559013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.559479 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.559480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:39.559537 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:39.560008 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:39.560013 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:39.561498 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:39.563251 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:39.563252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:39.563253 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:39.563458 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:39.563639 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:39.563640 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:39.563640 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:39.563712 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:39.563984 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:39.563985 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:39.564037 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 02:38:39.564109 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 02:38:39.564201 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 02:38:39.564318 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.556760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54050 (local address 127.0.98.60:34487)
0928 02:38:39.556805 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:39.556807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:39.556852 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:38:39.556879 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:39.556880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:39.556890 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:39.556942 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:39.557343 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.558103 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.559094 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.559404 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.559481 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:39.561589 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:39.561591 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:39.561591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:39.561598 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:39.563178 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:39.563530 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:39.563531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:39.563532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:39.563589 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:39.563782 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:39.563783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:39.563784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:39.563916 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:39.563979 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:39.564144 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 02:38:39.564220 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 02:38:39.566406 22653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e39df79b28b44049c020a7011c96682: Bootstrap starting.
I20260928 02:38:39.567164 22653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e39df79b28b44049c020a7011c96682: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:39.567448 22653 log.cc:824] T 00000000000000000000000000000000 P 4e39df79b28b44049c020a7011c96682: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:39.568336 22653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e39df79b28b44049c020a7011c96682: No bootstrap required, opened a new log
I20260928 02:38:39.569698 22653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e39df79b28b44049c020a7011c96682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39df79b28b44049c020a7011c96682" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39673 } } peers { permanent_uuid: "932214c249b94adcbf2480b93ec96202" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46615 } } peers { permanent_uuid: "16bf871a38c64ef191a63926ca7c697c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34487 } }
I20260928 02:38:39.569834 22653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e39df79b28b44049c020a7011c96682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e39df79b28b44049c020a7011c96682, State: Initialized, Role: FOLLOWER
I20260928 02:38:39.569908 22653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e39df79b28b44049c020a7011c96682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39df79b28b44049c020a7011c96682" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39673 } } peers { permanent_uuid: "932214c249b94adcbf2480b93ec96202" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46615 } } peers { permanent_uuid: "16bf871a38c64ef191a63926ca7c697c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34487 } }
I20260928 02:38:39.570182 22839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e39df79b28b44049c020a7011c96682 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39df79b28b44049c020a7011c96682" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39673 } } peers { permanent_uuid: "932214c249b94adcbf2480b93ec96202" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46615 } } peers { permanent_uuid: "16bf871a38c64ef191a63926ca7c697c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34487 } } }
I20260928 02:38:39.570218 22627 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e39df79b28b44049c020a7011c96682 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 02:38:39.570327 22839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e39df79b28b44049c020a7011c96682 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:38:39.570393 22627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e39df79b28b44049c020a7011c96682 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:39.571419 22627 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e39df79b28b44049c020a7011c96682 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39df79b28b44049c020a7011c96682" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39673 } } peers { permanent_uuid: "932214c249b94adcbf2480b93ec96202" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46615 } } peers { permanent_uuid: "16bf871a38c64ef191a63926ca7c697c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34487 } }
I20260928 02:38:39.571638 22627 leader_election.cc:288] T 00000000000000000000000000000000 P 4e39df79b28b44049c020a7011c96682 [CANDIDATE]: Term 1 election: Requested vote from peers 932214c249b94adcbf2480b93ec96202 (127.0.98.61:46615), 16bf871a38c64ef191a63926ca7c697c (127.0.98.60:34487)
I20260928 02:38:39.572986 22702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e39df79b28b44049c020a7011c96682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "932214c249b94adcbf2480b93ec96202"
I20260928 02:38:39.573102 22702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 932214c249b94adcbf2480b93ec96202 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:39.573896 22702 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 932214c249b94adcbf2480b93ec96202 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e39df79b28b44049c020a7011c96682 in term 1.
I20260928 02:38:39.574131 22777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e39df79b28b44049c020a7011c96682" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "16bf871a38c64ef191a63926ca7c697c"
I20260928 02:38:39.574244 22777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16bf871a38c64ef191a63926ca7c697c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:39.575042 22777 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 16bf871a38c64ef191a63926ca7c697c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e39df79b28b44049c020a7011c96682 in term 1.
I20260928 02:38:39.575446 22604 leader_election.cc:302] T 00000000000000000000000000000000 P 4e39df79b28b44049c020a7011c96682 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e39df79b28b44049c020a7011c96682, 932214c249b94adcbf2480b93ec96202; no voters:
I20260928 02:38:39.575598 22839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e39df79b28b44049c020a7011c96682 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:39.575760 22839 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e39df79b28b44049c020a7011c96682 [term 1 LEADER]: Becoming Leader. State: Replica: 4e39df79b28b44049c020a7011c96682, State: Running, Role: LEADER
I20260928 02:38:39.575841 22839 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e39df79b28b44049c020a7011c96682 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39df79b28b44049c020a7011c96682" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39673 } } peers { permanent_uuid: "932214c249b94adcbf2480b93ec96202" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46615 } } peers { permanent_uuid: "16bf871a38c64ef191a63926ca7c697c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34487 } }
I20260928 02:38:39.576208 22839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e39df79b28b44049c020a7011c96682 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e39df79b28b44049c020a7011c96682. Latest consensus state: current_term: 1 leader_uuid: "4e39df79b28b44049c020a7011c96682" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39df79b28b44049c020a7011c96682" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39673 } } peers { permanent_uuid: "932214c249b94adcbf2480b93ec96202" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46615 } } peers { permanent_uuid: "16bf871a38c64ef191a63926ca7c697c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34487 } } }
I20260928 02:38:39.576290 22839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e39df79b28b44049c020a7011c96682 [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:39.570192 22653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e39df79b28b44049c020a7011c96682 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 02:38:39.578975 22858 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e39df79b28b44049c020a7011c96682: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:38:39.579073 22858 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:38:39.582123 22847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:38:39.582298 22847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:38:39.584016 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.572725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58142 (local address 127.0.98.62:39673)
0928 02:38:39.572765 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:39.572767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:39.573037 (+ 270us) server_negotiation.cc:407] Connection header received
0928 02:38:39.573129 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:39.573130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:39.573141 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:39.573168 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:39.573925 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.574691 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.575748 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.576110 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.576149 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:39.577722 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:39.577723 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:39.577723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:39.577730 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:39.581768 (+ 4038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:39.583201 (+ 1433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:39.583202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:39.583203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:39.583273 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:39.583465 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:39.583466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:39.583467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:39.583586 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:39.583653 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:39.583847 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 02:38:39.583926 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:38:39.588012 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.578013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50418 (local address 127.0.98.61:46615)
0928 02:38:39.578049 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:39.578051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:39.579327 (+ 1276us) server_negotiation.cc:407] Connection header received
0928 02:38:39.579416 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:39.579417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:39.579427 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:39.579453 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:39.580187 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.580882 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.581936 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.582246 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.582302 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:39.585357 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:39.585358 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:39.585359 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:39.585365 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:39.586873 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:39.587252 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:39.587253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:39.587254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:39.587306 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:39.587486 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:39.587487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:39.587487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:39.587602 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:39.587667 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:39.587850 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 02:38:39.587922 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:38:39.589989 22702 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 932214c249b94adcbf2480b93ec96202 [term 1 FOLLOWER]: Refusing update from remote peer 4e39df79b28b44049c020a7011c96682: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:38:39.589994 22777 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 16bf871a38c64ef191a63926ca7c697c [term 1 FOLLOWER]: Refusing update from remote peer 4e39df79b28b44049c020a7011c96682: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:38:39.590308 22845 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e39df79b28b44049c020a7011c96682 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16bf871a38c64ef191a63926ca7c697c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:38:39.590418 22839 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e39df79b28b44049c020a7011c96682 [LEADER]: Connected to new peer: Peer: permanent_uuid: "932214c249b94adcbf2480b93ec96202" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:38:39.591645 22811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932214c249b94adcbf2480b93ec96202 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e39df79b28b44049c020a7011c96682. Latest consensus state: current_term: 1 leader_uuid: "4e39df79b28b44049c020a7011c96682" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39df79b28b44049c020a7011c96682" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39673 } } peers { permanent_uuid: "932214c249b94adcbf2480b93ec96202" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46615 } } peers { permanent_uuid: "16bf871a38c64ef191a63926ca7c697c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34487 } } }
I20260928 02:38:39.591725 22811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932214c249b94adcbf2480b93ec96202 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:38:39.591787 22825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16bf871a38c64ef191a63926ca7c697c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e39df79b28b44049c020a7011c96682. Latest consensus state: current_term: 1 leader_uuid: "4e39df79b28b44049c020a7011c96682" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39df79b28b44049c020a7011c96682" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39673 } } peers { permanent_uuid: "932214c249b94adcbf2480b93ec96202" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46615 } } peers { permanent_uuid: "16bf871a38c64ef191a63926ca7c697c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34487 } } }
I20260928 02:38:39.591858 22825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16bf871a38c64ef191a63926ca7c697c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:38:39.595580 22845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e39df79b28b44049c020a7011c96682 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e39df79b28b44049c020a7011c96682" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39df79b28b44049c020a7011c96682" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39673 } } peers { permanent_uuid: "932214c249b94adcbf2480b93ec96202" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46615 } } peers { permanent_uuid: "16bf871a38c64ef191a63926ca7c697c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34487 } } }
I20260928 02:38:39.595716 22845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e39df79b28b44049c020a7011c96682 [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:39.595856 22864 mvcc.cc:205] Tried to move back new op lower bound from 7334146537838628864 to 7334146537783398400. Current Snapshot: MvccSnapshot[applied={T|T < 7334146537838628864}]
I20260928 02:38:39.596040 22825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16bf871a38c64ef191a63926ca7c697c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e39df79b28b44049c020a7011c96682" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39df79b28b44049c020a7011c96682" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39673 } } peers { permanent_uuid: "932214c249b94adcbf2480b93ec96202" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46615 } } peers { permanent_uuid: "16bf871a38c64ef191a63926ca7c697c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34487 } } }
I20260928 02:38:39.596123 22825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16bf871a38c64ef191a63926ca7c697c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:38:39.597553 22839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e39df79b28b44049c020a7011c96682 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e39df79b28b44049c020a7011c96682" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39df79b28b44049c020a7011c96682" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39673 } } peers { permanent_uuid: "932214c249b94adcbf2480b93ec96202" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46615 } } peers { permanent_uuid: "16bf871a38c64ef191a63926ca7c697c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34487 } } }
I20260928 02:38:39.597649 22839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e39df79b28b44049c020a7011c96682 [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:39.597991 22811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932214c249b94adcbf2480b93ec96202 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e39df79b28b44049c020a7011c96682" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e39df79b28b44049c020a7011c96682" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39673 } } peers { permanent_uuid: "932214c249b94adcbf2480b93ec96202" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46615 } } peers { permanent_uuid: "16bf871a38c64ef191a63926ca7c697c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34487 } } }
I20260928 02:38:39.598071 22811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932214c249b94adcbf2480b93ec96202 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:38:39.598713 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:39.578311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54066 (local address 127.0.98.60:34487)
0928 02:38:39.578353 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:39.578355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:39.581825 (+ 3470us) server_negotiation.cc:407] Connection header received
0928 02:38:39.581914 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:39.581915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:39.581925 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:39.581953 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:39.582537 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.583190 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.593409 (+ 10219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:39.594069 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:39.594219 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:39.595685 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:39.595687 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:39.595687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:39.595694 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:39.597373 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:39.597725 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:39.597726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:39.597727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:39.597779 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:39.597950 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:39.597951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:39.597951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:39.598068 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:39.598144 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:39.598557 (+ 413us) server_negotiation.cc:299] Negotiation successful
0928 02:38:39.598625 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:38:39.598711 22863 mvcc.cc:205] Tried to move back new op lower bound from 7334146537838628864 to 7334146537783398400. Current Snapshot: MvccSnapshot[applied={T|T < 7334146537838628864}]
I20260928 02:38:39.605947 22847 catalog_manager.cc:1347] Generated new cluster ID: 9471b8400cbd4982b66c52a1e11a1ef6
I20260928 02:38:39.605998 22847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:38:39.620891 22847 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:38:39.621639 22847 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:38:39.635337 22847 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e39df79b28b44049c020a7011c96682: Generated new TSK 0
I20260928 02:38:39.635558 22847 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:38:39.656280 22732 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:38:39.654975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50428 (local address 127.0.98.61:46615)
0928 02:38:39.655016 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:39.655018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:39.655608 (+ 590us) server_negotiation.cc:407] Connection header received
0928 02:38:39.655745 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:39.655747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:39.655759 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:39.655790 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:39.656203 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50428: BlockingRecv error: recv got EOF from 127.0.0.1:50428 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 02:38:39.657058 22807 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:38:39.655310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54072 (local address 127.0.98.60:34487)
0928 02:38:39.655351 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:39.655352 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:39.656479 (+ 1127us) server_negotiation.cc:407] Connection header received
0928 02:38:39.656642 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:39.656643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:39.656654 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:39.656679 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:39.656998 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54072: BlockingRecv error: recv got EOF from 127.0.0.1:54072 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 02:38:39.658017 22810 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:38:39.657471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58154 (local address 127.0.98.62:39673)
0928 02:38:39.657521 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:39.657523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:39.657554 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:38:39.657637 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:39.657638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:39.657647 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:39.657673 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:39.657961 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58154: BlockingRecv error: recv got EOF from 127.0.0.1:58154 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 02:38:39.660162 22732 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:38:39.659402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50432 (local address 127.0.98.61:46615)
0928 02:38:39.659437 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:39.659439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:39.659663 (+ 224us) server_negotiation.cc:407] Connection header received
0928 02:38:39.659758 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:39.659759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:39.659775 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:39.659803 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:39.660118 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50432: BlockingRecv error: recv got EOF from 127.0.0.1:50432 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 02:38:39.662951 22732 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:38:39.662226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50436 (local address 127.0.98.61:46615)
0928 02:38:39.662274 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:39.662276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:39.662502 (+ 226us) server_negotiation.cc:407] Connection header received
0928 02:38:39.662589 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:39.662590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:39.662599 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:39.662626 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:39.662909 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50436: BlockingRecv error: recv got EOF from 127.0.0.1:50436 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260928 02:38:39.666656 22732 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:38:39.665956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50448 (local address 127.0.98.61:46615)
0928 02:38:39.665989 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:39.665991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:39.666220 (+ 229us) server_negotiation.cc:407] Connection header received
0928 02:38:39.666322 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:39.666323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:39.666332 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:39.666356 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:39.666615 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50448: BlockingRecv error: recv got EOF from 127.0.0.1:50448 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 02:38:39.672389 22732 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:38:39.671637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50456 (local address 127.0.98.61:46615)
0928 02:38:39.671672 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:39.671674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:39.671899 (+ 225us) server_negotiation.cc:407] Connection header received
0928 02:38:39.671985 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:39.671985 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:39.671995 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:39.672022 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:39.672345 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50456: BlockingRecv error: recv got EOF from 127.0.0.1:50456 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 02:38:39.682652 22732 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:38:39.681671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50464 (local address 127.0.98.61:46615)
0928 02:38:39.681720 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:39.681722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:39.682032 (+ 310us) server_negotiation.cc:407] Connection header received
0928 02:38:39.682135 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:39.682136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:39.682149 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:39.682179 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:39.682592 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50464: BlockingRecv error: recv got EOF from 127.0.0.1:50464 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 02:38:39.700474 22732 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:38:39.699831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50480 (local address 127.0.98.61:46615)
0928 02:38:39.699883 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:39.699886 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:39.699972 (+ 86us) server_negotiation.cc:407] Connection header received
0928 02:38:39.700005 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:39.700006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:39.700018 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:39.700100 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:39.700414 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50480: BlockingRecv error: recv got EOF from 127.0.0.1:50480 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 02:38:39.734258 22732 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:38:39.733495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50496 (local address 127.0.98.61:46615)
0928 02:38:39.733557 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:39.733559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:39.733682 (+ 123us) server_negotiation.cc:407] Connection header received
0928 02:38:39.733757 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:39.733758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:39.733768 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:39.733818 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:39.734184 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50496: BlockingRecv error: recv got EOF from 127.0.0.1:50496 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260928 02:38:39.799968 22732 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:38:39.799330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50506 (local address 127.0.98.61:46615)
0928 02:38:39.799391 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:39.799393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:39.799471 (+ 78us) server_negotiation.cc:407] Connection header received
0928 02:38:39.799535 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:39.799536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:39.799546 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:39.799585 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:39.799910 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50506: BlockingRecv error: recv got EOF from 127.0.0.1:50506 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 02:38:39.929646 22732 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:38:39.928988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50522 (local address 127.0.98.61:46615)
0928 02:38:39.929060 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:39.929062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:39.929119 (+ 57us) server_negotiation.cc:407] Connection header received
0928 02:38:39.929191 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:39.929192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:39.929204 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:39.929277 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:39.929585 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50522: BlockingRecv error: recv got EOF from 127.0.0.1:50522 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 02:38:40.187338 22732 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:38:40.186728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50538 (local address 127.0.98.61:46615)
0928 02:38:40.186798 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:40.186800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:40.186809 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:38:40.186878 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:40.186879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:40.186891 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:40.186935 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:40.187275 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50538: BlockingRecv error: recv got EOF from 127.0.0.1:50538 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:38:40.510710 22822 catalog_manager.cc:1259] Loaded cluster ID: 9471b8400cbd4982b66c52a1e11a1ef6
I20260928 02:38:40.510774 22822 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 932214c249b94adcbf2480b93ec96202: loading cluster ID for follower catalog manager: success
I20260928 02:38:40.511523 22822 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 932214c249b94adcbf2480b93ec96202: acquiring CA information for follower catalog manager: success
I20260928 02:38:40.511770 22822 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 932214c249b94adcbf2480b93ec96202: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:38:40.529330 22836 catalog_manager.cc:1259] Loaded cluster ID: 9471b8400cbd4982b66c52a1e11a1ef6
I20260928 02:38:40.529384 22836 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 16bf871a38c64ef191a63926ca7c697c: loading cluster ID for follower catalog manager: success
I20260928 02:38:40.530131 22836 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 16bf871a38c64ef191a63926ca7c697c: acquiring CA information for follower catalog manager: success
I20260928 02:38:40.530370 22836 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 16bf871a38c64ef191a63926ca7c697c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:38:40.702857 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:40.700435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50540 (local address 127.0.98.61:46615)
0928 02:38:40.700530 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:40.700532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:40.700599 (+ 67us) server_negotiation.cc:407] Connection header received
0928 02:38:40.700623 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:40.700624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:40.700672 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:40.700745 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:38:40.700865 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:40.701380 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:40.702100 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:40.702299 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:40.702351 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:40.702511 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:38:40.702648 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:38:40.702719 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:40.702750 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 02:38:40.702771 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":47,"threads_started":1}
I20260928 02:38:40.706739 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:40.704640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54080 (local address 127.0.98.60:34487)
0928 02:38:40.704745 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:40.704747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:40.704755 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:38:40.704791 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:40.704792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:40.704825 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:40.704893 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:38:40.704965 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:40.705471 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:40.706132 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:40.706326 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:40.706374 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:40.706475 (+ 101us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:38:40.706571 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:38:40.706630 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:40.706643 (+ 13us) server_negotiation.cc:299] Negotiation successful
0928 02:38:40.706659 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":48,"threads_started":1}
I20260928 02:38:40.710983 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:40.708397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54084 (local address 127.0.98.60:34487)
0928 02:38:40.708439 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:40.708441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:40.708451 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:38:40.708542 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:40.708543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:40.708554 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:40.708628 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:38:40.708772 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:40.709286 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:40.710328 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:40.710604 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:40.710662 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:40.710772 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:38:40.710799 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:38:40.710841 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:40.710900 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 02:38:40.710918 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 02:38:40.711453 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:40.708226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50550 (local address 127.0.98.61:46615)
0928 02:38:40.708310 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:40.708311 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:40.708434 (+ 123us) server_negotiation.cc:407] Connection header received
0928 02:38:40.708501 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:40.708502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:40.708513 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:40.708556 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:38:40.708907 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:40.709707 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:40.710618 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:40.710900 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:40.710953 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:40.711224 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:38:40.711256 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:38:40.711301 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:40.711368 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 02:38:40.711387 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:38:40.711612 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:40.708214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58170 (local address 127.0.98.62:39673)
0928 02:38:40.708503 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:40.708505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:40.708511 (+ 6us) server_negotiation.cc:407] Connection header received
0928 02:38:40.708553 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:40.708553 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:40.708600 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:40.708667 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:38:40.708874 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:40.709722 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:40.710581 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:40.710873 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:40.710927 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:40.711202 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:38:40.711298 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:38:40.711349 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:40.711419 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 02:38:40.711439 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":60,"threads_started":1}
I20260928 02:38:40.712997 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 22585
I20260928 02:38:40.718214 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 22660
I20260928 02:38:40.722488 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 22735
2026-09-28T02:38:40Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2672 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:38:40 dist-test-slave-fz47 krb5kdc[22950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:38:40 dist-test-slave-fz47 krb5kdc[22950](info): set up 2 sockets
Sep 28 02:38:40 dist-test-slave-fz47 krb5kdc[22950](info): commencing operation
krb5kdc: starting...
W20260928 02:38:41.677350 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/krb5kdc/test-user.keytab.
Sep 28 02:38:41 dist-test-slave-fz47 krb5kdc[22950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:38:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:38:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 02:38:41.787982 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37197
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37197,127.0.98.61:41365,127.0.98.60:36541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:38:41.864681 22963 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:38:41.864884 22963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:41.864905 22963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:41.864929 22963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:41.866039 22963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:41.866079 22963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:41.866097 22963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:41.866112 22963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:41.866127 22963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:41.867566 22963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37197,127.0.98.61:41365,127.0.98.60:36541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:41.867810 22963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:41.868010 22963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:41.872620 22969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:41.872648 22968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:41.872738 22963 server_base.cc:1034] running on GCE node
W20260928 02:38:41.872646 22971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:41.873070 22963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:41.873353 22963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:41.874517 22963 hybrid_clock.cc:648] HybridClock initialized: now 1790563121874502 us; error 29 us; skew 500 ppm
Sep 28 02:38:41 dist-test-slave-fz47 krb5kdc[22950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:41.881531 22963 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 02:38:41.882001 22963 webserver.cc:460] Webserver started at http://127.0.98.62:38759/ using document root <none> and password file <none>
I20260928 02:38:41.882215 22963 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:41.882262 22963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:41.882388 22963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:41.883419 22963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-0/data/instance:
uuid: "389bf0585a2c4187bc2feb7f262f29a6"
format_stamp: "Formatted at 2026-09-28 02:38:41 on dist-test-slave-fz47"
server_key: "8585f078c0fb21f8a61051767c9c1a9e"
server_key_iv: "4470db2a3531fd48c8bf98567478bbb2"
server_key_version: "encryptionkey@0"
I20260928 02:38:41.883738 22963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-0/wal/instance:
uuid: "389bf0585a2c4187bc2feb7f262f29a6"
format_stamp: "Formatted at 2026-09-28 02:38:41 on dist-test-slave-fz47"
server_key: "8585f078c0fb21f8a61051767c9c1a9e"
server_key_iv: "4470db2a3531fd48c8bf98567478bbb2"
server_key_version: "encryptionkey@0"
I20260928 02:38:41.884938 22963 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 02:38:41.885673 22978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:41.885826 22963 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:41.885907 22963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-0/wal
uuid: "389bf0585a2c4187bc2feb7f262f29a6"
format_stamp: "Formatted at 2026-09-28 02:38:41 on dist-test-slave-fz47"
server_key: "8585f078c0fb21f8a61051767c9c1a9e"
server_key_iv: "4470db2a3531fd48c8bf98567478bbb2"
server_key_version: "encryptionkey@0"
I20260928 02:38:41.885968 22963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:41.909116 22963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:41.910914 22963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:41.911080 22963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:41.914860 22963 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37197
I20260928 02:38:41.914911 23030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37197 every 8 connection(s)
I20260928 02:38:41.915227 22963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-0/data/info.pb
I20260928 02:38:41.916154 23031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:41.917719 23031 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37197 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:38:41.922235 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 22963
I20260928 02:38:41.922315 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-0/wal/instance
I20260928 02:38:41.922480 392 external_mini_cluster.cc:1428] Setting key afafda52ead10bd28c3a7b5c56b630b4
Sep 28 02:38:41 dist-test-slave-fz47 krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:38:41.926578 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:41.918378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39812 (local address 127.0.98.62:37197)
0928 02:38:41.918529 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:41.918533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:41.918625 (+ 92us) server_negotiation.cc:407] Connection header received
0928 02:38:41.919089 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:41.919090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:41.919108 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:41.919180 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:41.919689 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:41.920472 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:41.921291 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:41.921588 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:41.921683 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:41.923597 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:41.923604 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:41.923604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:41.923610 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:41.925035 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:41.925497 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:41.925499 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:41.925500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:41.925574 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:41.925824 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:41.925825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:41.925825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:41.926013 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:41.926071 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:41.926300 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 02:38:41.926425 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":121,"thread_start_us":70,"threads_started":1}
I20260928 02:38:41.926672 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:41.918337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37197 (local address 127.0.0.1:39812)
0928 02:38:41.918489 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:41.918524 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:41.918972 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:41.919215 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:41.919217 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:41.919451 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:41.919602 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:41.919606 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:41.920545 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:41.920545 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:41.921208 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:41.921210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:41.921688 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:41.921689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:41.921742 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:41.922194 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:41.922201 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:41.923513 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:41.925121 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:41.925128 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:41.925130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:41.925417 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:41.925662 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:41.925663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:41.925663 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:41.925766 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:41.926082 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:41.926090 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:41.926177 (+ 87us) client_negotiation.cc:769] Sending connection context
0928 02:38:41.926281 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 02:38:41.926502 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":71,"thread_start_us":67,"threads_started":1}
I20260928 02:38:41.927419 23031 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41365 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:38:41.927755 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:41.927634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41365 (local address 127.0.0.1:60844)
0928 02:38:41.927686 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:41.927733 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 02:38:41.927896 22981 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41365: connect: Connection refused (error 111)
I20260928 02:38:41.928277 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:41.928178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41365 (local address 127.0.0.1:60854)
0928 02:38:41.928229 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:41.928255 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 02:38:41.928392 23031 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41365: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41365: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260928 02:38:41.945798 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41365
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37197,127.0.98.61:41365,127.0.98.60:36541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:38:41.977597 23031 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41365 } attempt: 1
I20260928 02:38:41.978118 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:41.977925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41365 (local address 127.0.0.1:60864)
0928 02:38:41.977995 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:41.978048 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 02:38:41.978533 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:41.978410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41365 (local address 127.0.0.1:60866)
0928 02:38:41.978483 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:41.978510 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 02:38:41.978638 23031 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41365: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41365: connect: Connection refused (error 111)
W20260928 02:38:42.023677 23038 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:38:42.023876 23038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:42.023897 23038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:42.023921 23038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:42.024992 23038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:42.025029 23038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:42.025046 23038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:42.025063 23038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:42.025077 23038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:42.026502 23038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37197,127.0.98.61:41365,127.0.98.60:36541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:42.026741 23038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:42.026940 23038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:42.031391 23043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:42.031436 23038 server_base.cc:1034] running on GCE node
W20260928 02:38:42.031461 23046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:42.031546 23044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:42.032004 23038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:42.032277 23038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:42.033423 23038 hybrid_clock.cc:648] HybridClock initialized: now 1790563122033405 us; error 32 us; skew 500 ppm
Sep 28 02:38:42 dist-test-slave-fz47 krb5kdc[22950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:42.035815 23038 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260928 02:38:42.036221 23038 webserver.cc:460] Webserver started at http://127.0.98.61:40951/ using document root <none> and password file <none>
I20260928 02:38:42.036414 23038 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:42.036466 23038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:42.036598 23038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:42.037726 23038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-1/data/instance:
uuid: "9581fe9a54934139a6aee3eefa77618f"
format_stamp: "Formatted at 2026-09-28 02:38:42 on dist-test-slave-fz47"
server_key: "b98bf69a5e75abfc82132ece636786ac"
server_key_iv: "4ad16e9ead447fce175d6ae4c9572a2b"
server_key_version: "encryptionkey@0"
I20260928 02:38:42.038043 23038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-1/wal/instance:
uuid: "9581fe9a54934139a6aee3eefa77618f"
format_stamp: "Formatted at 2026-09-28 02:38:42 on dist-test-slave-fz47"
server_key: "b98bf69a5e75abfc82132ece636786ac"
server_key_iv: "4ad16e9ead447fce175d6ae4c9572a2b"
server_key_version: "encryptionkey@0"
I20260928 02:38:42.039247 23038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:38:42.040016 23053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:42.040191 23038 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:42.040272 23038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-1/wal
uuid: "9581fe9a54934139a6aee3eefa77618f"
format_stamp: "Formatted at 2026-09-28 02:38:42 on dist-test-slave-fz47"
server_key: "b98bf69a5e75abfc82132ece636786ac"
server_key_iv: "4ad16e9ead447fce175d6ae4c9572a2b"
server_key_version: "encryptionkey@0"
I20260928 02:38:42.040339 23038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:42.046797 23031 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41365 } attempt: 2
I20260928 02:38:42.047169 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:42.047059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41365 (local address 127.0.0.1:60878)
0928 02:38:42.047113 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:42.047143 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 02:38:42.047499 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:42.047409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41365 (local address 127.0.0.1:60890)
0928 02:38:42.047452 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:42.047477 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 02:38:42.047588 23031 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:41365: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41365: connect: Connection refused (error 111)
I20260928 02:38:42.051923 23038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:42.053606 23038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:42.053742 23038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:42.057284 23038 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41365
I20260928 02:38:42.057329 23105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41365 every 8 connection(s)
I20260928 02:38:42.057647 23038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-1/data/info.pb
I20260928 02:38:42.058533 23106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:42.059868 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 23038
I20260928 02:38:42.059949 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-1/wal/instance
I20260928 02:38:42.060128 392 external_mini_cluster.cc:1428] Setting key 93a1dcb0745f81d6a83904e4494dac86
I20260928 02:38:42.059993 23106 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37197 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:38:42 dist-test-slave-fz47 krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260928 02:38:42.069063 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:42.060650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39822 (local address 127.0.98.62:37197)
0928 02:38:42.060715 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:42.060718 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:42.060891 (+ 173us) server_negotiation.cc:407] Connection header received
0928 02:38:42.061518 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:42.061519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:42.061533 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:42.061600 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:42.062159 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.062660 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.064031 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.064229 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.064279 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:42.066476 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:42.066478 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:42.066478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:42.066485 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:42.067764 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:42.068155 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:42.068157 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:42.068157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:42.068207 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:42.068425 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:42.068426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:42.068426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:42.068526 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:42.068581 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:42.068897 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 02:38:42.068973 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:38:42.069283 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:42.060600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37197 (local address 127.0.0.1:39822)
0928 02:38:42.060760 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:42.060806 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:42.061420 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:42.061659 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:42.061662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:42.061890 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:42.062069 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:42.062074 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.062752 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.062753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:42.063950 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:42.063953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.064320 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.064321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:42.064472 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:42.064943 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:42.064953 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:42.066381 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:42.067846 (+ 1465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:42.067883 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:42.067885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:42.068082 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:42.068269 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:42.068269 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:42.068270 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:42.068364 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:42.068587 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:42.068593 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:42.068772 (+ 179us) client_negotiation.cc:769] Sending connection context
0928 02:38:42.068858 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 02:38:42.069000 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260928 02:38:42.069814 23106 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41365 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:38:42 dist-test-slave-fz47 krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 02:38:42.076900 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:42.070021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41365 (local address 127.0.0.1:60904)
0928 02:38:42.070069 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:42.070080 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:42.070123 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:42.070328 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:42.070329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:42.070459 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:42.070511 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:42.070513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.071688 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.071689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:42.072412 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:42.072414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.072853 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.072854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:42.072929 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:42.073251 (+ 322us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:42.073271 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:42.074431 (+ 1160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:42.075950 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:42.075951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:42.075951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:42.076094 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:42.076273 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:42.076274 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:42.076274 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:42.076325 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:42.076572 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:42.076573 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:42.076609 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 02:38:42.076668 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 02:38:42.076792 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 02:38:42.076958 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:42.070056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60904 (local address 127.0.98.61:41365)
0928 02:38:42.070200 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:42.070204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:42.070222 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:38:42.070242 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:42.070244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:42.070256 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:42.070314 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:42.070582 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.071601 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.072482 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.072780 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.072827 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:42.074508 (+ 1681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:42.074515 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:42.074515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:42.074521 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:42.075868 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:42.076164 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:42.076167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:42.076167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:42.076214 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:42.076379 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:42.076380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:42.076380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:42.076507 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:42.076574 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:42.076718 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 02:38:42.076840 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260928 02:38:42.077608 23106 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36541 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:38:42.077942 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:42.077808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36541 (local address 127.0.0.1:33336)
0928 02:38:42.077871 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:42.077923 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 02:38:42.078064 23057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36541: connect: Connection refused (error 111)
I20260928 02:38:42.078423 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:42.078337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36541 (local address 127.0.0.1:33348)
0928 02:38:42.078380 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:42.078402 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 02:38:42.078514 23106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36541: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36541: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260928 02:38:42.084337 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskyt1071/build/release/bin/kudu
/tmp/dist-test-taskyt1071/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36541
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37197,127.0.98.61:41365,127.0.98.60:36541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:38:42.127733 23106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36541 } attempt: 1
I20260928 02:38:42.128232 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:42.128085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36541 (local address 127.0.0.1:33360)
0928 02:38:42.128163 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:42.128203 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 02:38:42.128585 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:42.128489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36541 (local address 127.0.0.1:33376)
0928 02:38:42.128540 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:42.128564 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 02:38:42.128680 23106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36541: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36541: connect: Connection refused (error 111)
I20260928 02:38:42.138731 23031 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41365 } attempt: 3
Sep 28 02:38:42 dist-test-slave-fz47 krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 02:38:42.145824 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:42.139112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60908 (local address 127.0.98.61:41365)
0928 02:38:42.139172 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:42.139174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:42.139184 (+ 10us) server_negotiation.cc:407] Connection header received
0928 02:38:42.139229 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:42.139229 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:42.139240 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:42.139288 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:42.139725 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.140232 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.141130 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.141426 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.141488 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:42.143496 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:42.143497 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:42.143497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:42.143503 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:42.144785 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:42.145152 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:42.145153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:42.145153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:42.145201 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:42.145410 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:42.145411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:42.145411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:42.145507 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:42.145555 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:42.145704 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 02:38:42.145765 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 02:38:42.145887 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:42.139064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41365 (local address 127.0.0.1:60908)
0928 02:38:42.139103 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:42.139114 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:42.139170 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:42.139301 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:42.139302 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:42.139472 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:42.139566 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:42.139569 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.140330 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.140331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:42.141023 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:42.141026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.141492 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.141493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:42.141563 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:42.141994 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:42.141999 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:42.143417 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:42.144866 (+ 1449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:42.144867 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:42.144868 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:42.145080 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:42.145281 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:42.145282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:42.145283 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:42.145350 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:42.145565 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:42.145566 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:42.145613 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 02:38:42.145691 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 02:38:42.145784 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 02:38:42.146250 23031 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36541 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:38:42.146570 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:42.146458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36541 (local address 127.0.0.1:33388)
0928 02:38:42.146508 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:42.146545 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 02:38:42.146904 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:42.146843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36541 (local address 127.0.0.1:33390)
0928 02:38:42.146865 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:42.146888 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260928 02:38:42.146991 23031 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36541: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36541: connect: Connection refused (error 111)
W20260928 02:38:42.163553 23113 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:38:42.163759 23113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:38:42.163775 23113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:38:42.163793 23113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:38:42.164921 23113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:38:42.164963 23113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:38:42.164975 23113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:38:42.164986 23113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:38:42.164996 23113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:38:42.166467 23113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37197,127.0.98.61:41365,127.0.98.60:36541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 01:48:29 UTC on ccycloud.quasar-jjsqvu.root.comops.site
build id 4676525
I20260928 02:38:42.166687 23113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:38:42.166889 23113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:38:42.171283 23121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:42.171443 23113 server_base.cc:1034] running on GCE node
W20260928 02:38:42.171320 23118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:38:42.171320 23119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:38:42.171756 23113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:38:42.172006 23113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:38:42.173156 23113 hybrid_clock.cc:648] HybridClock initialized: now 1790563122173146 us; error 29 us; skew 500 ppm
Sep 28 02:38:42 dist-test-slave-fz47 krb5kdc[22950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:38:42.175374 23113 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260928 02:38:42.175817 23113 webserver.cc:460] Webserver started at http://127.0.98.60:35335/ using document root <none> and password file <none>
I20260928 02:38:42.176003 23113 fs_manager.cc:359] Metadata directory not provided
I20260928 02:38:42.176048 23113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:38:42.176169 23113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:38:42.177135 23113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-2/data/instance:
uuid: "c80e2d2bef0b41d7a2c1cb8f86869144"
format_stamp: "Formatted at 2026-09-28 02:38:42 on dist-test-slave-fz47"
server_key: "32459425d6ec4f852718f1e4ab80acab"
server_key_iv: "7efa4a4dcbb9f7cb9ff78636821ccae9"
server_key_version: "encryptionkey@0"
I20260928 02:38:42.177477 23113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-2/wal/instance:
uuid: "c80e2d2bef0b41d7a2c1cb8f86869144"
format_stamp: "Formatted at 2026-09-28 02:38:42 on dist-test-slave-fz47"
server_key: "32459425d6ec4f852718f1e4ab80acab"
server_key_iv: "7efa4a4dcbb9f7cb9ff78636821ccae9"
server_key_version: "encryptionkey@0"
I20260928 02:38:42.178122 23031 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36541 } attempt: 1
I20260928 02:38:42.178493 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:42.178350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36541 (local address 127.0.0.1:33392)
0928 02:38:42.178417 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:42.178457 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 02:38:42.178807 23113 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:38:42.178860 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:42.178764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36541 (local address 127.0.0.1:33396)
0928 02:38:42.178809 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:42.178838 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 02:38:42.179014 23031 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36541: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36541: connect: Connection refused (error 111)
I20260928 02:38:42.179525 23128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:38:42.179668 23113 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 02:38:42.179746 23113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-2/wal
uuid: "c80e2d2bef0b41d7a2c1cb8f86869144"
format_stamp: "Formatted at 2026-09-28 02:38:42 on dist-test-slave-fz47"
server_key: "32459425d6ec4f852718f1e4ab80acab"
server_key_iv: "7efa4a4dcbb9f7cb9ff78636821ccae9"
server_key_version: "encryptionkey@0"
I20260928 02:38:42.179811 23113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:38:42.191051 23113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:38:42.192608 23113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:38:42.192719 23113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:38:42.196323 23113 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36541
I20260928 02:38:42.196353 23180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36541 every 8 connection(s)
I20260928 02:38:42.196657 23113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-2/data/info.pb
I20260928 02:38:42.196803 23106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36541 } attempt: 2
I20260928 02:38:42.197652 23181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:38:42.198807 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskyt1071/build/release/bin/kudu as pid 23113
I20260928 02:38:42.198907 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskyt1071/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790562862804661-392-0/minicluster-data/master-2/wal/instance
I20260928 02:38:42.199100 392 external_mini_cluster.cc:1428] Setting key 186fbe0ffcc665af0d32dbce81aa8681
I20260928 02:38:42.199258 23181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37197 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:38:42 dist-test-slave-fz47 krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 02:38:42 dist-test-slave-fz47 krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 28 02:38:42 dist-test-slave-fz47 krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 02:38:42.206858 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:42.197050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36541 (local address 127.0.0.1:33400)
0928 02:38:42.197121 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:42.197135 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:42.197196 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:42.198037 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:42.198038 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:42.198201 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:42.198252 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:42.198254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.199642 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.199643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:42.200381 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:42.200383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.201058 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.201059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:42.201115 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:42.201508 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:42.201513 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:42.202927 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:42.205603 (+ 2676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:42.205604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:42.205604 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:42.205819 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:42.206040 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:42.206041 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:42.206041 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:42.206105 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:42.206564 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:42.206565 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:42.206601 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 02:38:42.206667 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 02:38:42.206765 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 02:38:42.207104 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:42.197203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33400 (local address 127.0.98.60:36541)
0928 02:38:42.197395 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:42.197402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:42.197425 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:38:42.197845 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:42.197851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:42.197903 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:42.198018 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:42.198343 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.199553 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.200462 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.200982 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.201070 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:42.203004 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:42.203011 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:42.203011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:42.203022 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:42.205515 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:42.205884 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:42.205888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:42.205888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:42.205974 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:42.206179 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:42.206180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:42.206180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:42.206490 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:42.206587 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:42.206747 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 02:38:42.206899 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":111,"threads_started":1}
I20260928 02:38:42.207895 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:42.199881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37197 (local address 127.0.0.1:39834)
0928 02:38:42.200049 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:42.200097 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:42.200146 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:42.200306 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:42.200308 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:42.200594 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:42.200656 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:42.200658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.201550 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.201551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:42.202294 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:42.202297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.202817 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.202818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:42.202900 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:42.203253 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:42.203258 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:42.204636 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:42.206771 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:42.206777 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:42.206779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:42.207054 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:42.207280 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:42.207281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:42.207281 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:42.207335 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:42.207601 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:42.207608 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:42.207655 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 02:38:42.207737 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 02:38:42.207826 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":89,"threads_started":1}
I20260928 02:38:42.207949 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:42.200083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39834 (local address 127.0.98.62:37197)
0928 02:38:42.200126 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:42.200127 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:42.200135 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:38:42.200216 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:42.200217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:42.200226 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:42.200279 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:42.200743 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.201479 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.202373 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.202659 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.202724 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:42.204724 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:42.204725 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:42.204725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:42.204733 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:42.206693 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:42.207154 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:42.207156 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:42.207156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:42.207211 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:42.207400 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:42.207400 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:42.207401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:42.207519 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:42.207557 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:42.207796 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 02:38:42.207869 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 02:38:42.208292 23181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41365 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:38:42.208768 23106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9581fe9a54934139a6aee3eefa77618f: Bootstrap starting.
I20260928 02:38:42.209431 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:42.199444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33404 (local address 127.0.98.60:36541)
0928 02:38:42.199668 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:42.199669 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:42.199938 (+ 269us) server_negotiation.cc:407] Connection header received
0928 02:38:42.199960 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:42.199960 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:42.199970 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:42.200033 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:42.200752 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.201520 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.202311 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.202543 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.202595 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:42.205468 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:42.205469 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:42.205469 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:42.205475 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:42.207686 (+ 2211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:42.208381 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:42.208382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:42.208382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:42.208437 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:42.208647 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:42.208649 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:42.208649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:42.208767 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:42.208888 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:42.209229 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 02:38:42.209318 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":209,"thread_start_us":79,"threads_started":1}
I20260928 02:38:42.209710 23106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9581fe9a54934139a6aee3eefa77618f: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:42.210002 23106 log.cc:824] T 00000000000000000000000000000000 P 9581fe9a54934139a6aee3eefa77618f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:42.210775 23106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9581fe9a54934139a6aee3eefa77618f: No bootstrap required, opened a new log
Sep 28 02:38:42 dist-test-slave-fz47 krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 02:38:42.212776 23106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9581fe9a54934139a6aee3eefa77618f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389bf0585a2c4187bc2feb7f262f29a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37197 } } peers { permanent_uuid: "9581fe9a54934139a6aee3eefa77618f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41365 } } peers { permanent_uuid: "c80e2d2bef0b41d7a2c1cb8f86869144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36541 } }
I20260928 02:38:42.212909 23106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9581fe9a54934139a6aee3eefa77618f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9581fe9a54934139a6aee3eefa77618f, State: Initialized, Role: FOLLOWER
I20260928 02:38:42.212996 23106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9581fe9a54934139a6aee3eefa77618f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389bf0585a2c4187bc2feb7f262f29a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37197 } } peers { permanent_uuid: "9581fe9a54934139a6aee3eefa77618f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41365 } } peers { permanent_uuid: "c80e2d2bef0b41d7a2c1cb8f86869144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36541 } }
I20260928 02:38:42.213230 23188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9581fe9a54934139a6aee3eefa77618f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389bf0585a2c4187bc2feb7f262f29a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37197 } } peers { permanent_uuid: "9581fe9a54934139a6aee3eefa77618f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41365 } } peers { permanent_uuid: "c80e2d2bef0b41d7a2c1cb8f86869144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36541 } } }
I20260928 02:38:42.213325 23106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9581fe9a54934139a6aee3eefa77618f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:38:42.213364 23188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9581fe9a54934139a6aee3eefa77618f [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 02:38:42 dist-test-slave-fz47 krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 02:38:42.215303 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:42.208494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41365 (local address 127.0.0.1:60920)
0928 02:38:42.208641 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:42.208650 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:42.208690 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:42.208799 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:42.208800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:42.208923 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:42.208970 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:42.208971 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.209806 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.209807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:42.210340 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:42.210342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.210773 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.210774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:42.210828 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:42.211205 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:42.211209 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:42.212468 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:42.214209 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:42.214210 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:42.214211 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:42.214409 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:42.214590 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:42.214590 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:42.214591 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:42.214654 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:42.215009 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:42.215010 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:42.215050 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 02:38:42.215132 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 02:38:42.215216 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
W20260928 02:38:42.215173 23199 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9581fe9a54934139a6aee3eefa77618f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:38:42.215819 23199 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:38:42.215318 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:42.208578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60920 (local address 127.0.98.61:41365)
0928 02:38:42.208621 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:42.208623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:42.208724 (+ 101us) server_negotiation.cc:407] Connection header received
0928 02:38:42.208743 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:42.208743 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:42.208753 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:42.208796 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:42.209052 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.209724 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.210441 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.210701 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.210768 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:42.212555 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:42.212556 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:42.212557 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:42.212564 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:42.214112 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:42.214472 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:42.214473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:42.214473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:42.214530 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:42.214734 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:42.214734 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:42.214735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:42.214892 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:42.214954 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:42.215177 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 02:38:42.215240 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 02:38:42.216418 23181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36541 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:38:42.218142 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:42.210829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39842 (local address 127.0.98.62:37197)
0928 02:38:42.210865 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:42.210867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:42.210915 (+ 48us) server_negotiation.cc:407] Connection header received
0928 02:38:42.210933 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:42.210934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:42.210942 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:42.210991 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:42.211464 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.211888 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.212581 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.212844 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.212905 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:42.215286 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:42.215287 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:42.215287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:42.215293 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:42.216775 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:42.217122 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:42.217123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:42.217123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:42.217177 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:42.217388 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:42.217389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:42.217389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:42.217502 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:42.217624 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:42.217984 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 02:38:42.218057 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 02:38:42.216554 23038 master_runner.cc:186] Error getting master registration for 127.0.98.62:37197: OK, instance_id { permanent_uuid: "389bf0585a2c4187bc2feb7f262f29a6" instance_seqno: 1790563121913840 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:38:42.218943 23038 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 28 02:38:42 dist-test-slave-fz47 krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 02:38:42 dist-test-slave-fz47 krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 02:38:42.225291 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:42.216620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36541 (local address 127.0.0.1:33416)
0928 02:38:42.216827 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:42.216837 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:42.216873 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:42.219167 (+ 2294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:42.219167 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:42.219285 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:42.219309 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:42.219311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.219864 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.219865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:42.220508 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:42.220510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.220993 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.220994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:42.221044 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:42.221532 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:42.221536 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:42.222562 (+ 1026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:42.224199 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:42.224200 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:42.224200 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:42.224378 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:42.224593 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:42.224594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:42.224594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:42.224655 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:42.224970 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:42.224971 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:42.225011 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 02:38:42.225077 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 02:38:42.225153 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194}
I20260928 02:38:42.225366 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:42.216678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33416 (local address 127.0.98.60:36541)
0928 02:38:42.219006 (+ 2328us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:42.219008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:42.219016 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:38:42.219033 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:42.219034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:42.219042 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:42.219094 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:42.219373 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.219792 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.220593 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.220916 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.220988 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:42.222667 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:42.222669 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:42.222669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:42.222675 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:42.224118 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:42.224447 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:42.224448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:42.224448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:42.224503 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:42.224723 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:42.224724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:42.224724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:42.224891 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:42.224964 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:42.225142 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 02:38:42.225218 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2309}
I20260928 02:38:42.226178 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:42.218711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60936 (local address 127.0.98.61:41365)
0928 02:38:42.218741 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:42.218743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:42.218749 (+ 6us) server_negotiation.cc:407] Connection header received
0928 02:38:42.218767 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:42.218768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:42.218775 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:42.218830 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:42.219344 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.220020 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.220826 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.221048 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.221120 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:42.223542 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:42.223543 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:42.223543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:42.223549 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:42.224908 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:42.225245 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:42.225246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:42.225246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:42.225325 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:42.225676 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:42.225677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:42.225677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:42.225773 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:42.225897 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:42.225971 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 02:38:42.226031 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 02:38:42.226612 23181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c80e2d2bef0b41d7a2c1cb8f86869144: Bootstrap starting.
I20260928 02:38:42.227263 23181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c80e2d2bef0b41d7a2c1cb8f86869144: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:42.227541 23181 log.cc:824] T 00000000000000000000000000000000 P c80e2d2bef0b41d7a2c1cb8f86869144: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:42.228307 23181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c80e2d2bef0b41d7a2c1cb8f86869144: No bootstrap required, opened a new log
I20260928 02:38:42.228631 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:38:42.230011 23181 raft_consensus.cc:348] T 00000000000000000000000000000000 P c80e2d2bef0b41d7a2c1cb8f86869144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389bf0585a2c4187bc2feb7f262f29a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37197 } } peers { permanent_uuid: "9581fe9a54934139a6aee3eefa77618f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41365 } } peers { permanent_uuid: "c80e2d2bef0b41d7a2c1cb8f86869144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36541 } }
I20260928 02:38:42.230136 23181 raft_consensus.cc:729] T 00000000000000000000000000000000 P c80e2d2bef0b41d7a2c1cb8f86869144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c80e2d2bef0b41d7a2c1cb8f86869144, State: Initialized, Role: FOLLOWER
I20260928 02:38:42.230213 23181 consensus_queue.cc:260] T 00000000000000000000000000000000 P c80e2d2bef0b41d7a2c1cb8f86869144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389bf0585a2c4187bc2feb7f262f29a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37197 } } peers { permanent_uuid: "9581fe9a54934139a6aee3eefa77618f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41365 } } peers { permanent_uuid: "c80e2d2bef0b41d7a2c1cb8f86869144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36541 } }
I20260928 02:38:42.229489 23005 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "389bf0585a2c4187bc2feb7f262f29a6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39842
I20260928 02:38:42.230466 23202 sys_catalog.cc:455] T 00000000000000000000000000000000 P c80e2d2bef0b41d7a2c1cb8f86869144 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389bf0585a2c4187bc2feb7f262f29a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37197 } } peers { permanent_uuid: "9581fe9a54934139a6aee3eefa77618f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41365 } } peers { permanent_uuid: "c80e2d2bef0b41d7a2c1cb8f86869144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36541 } } }
I20260928 02:38:42.230521 23181 sys_catalog.cc:564] T 00000000000000000000000000000000 P c80e2d2bef0b41d7a2c1cb8f86869144 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:38:42.230571 23202 sys_catalog.cc:458] T 00000000000000000000000000000000 P c80e2d2bef0b41d7a2c1cb8f86869144 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 02:38:42.232472 23213 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c80e2d2bef0b41d7a2c1cb8f86869144: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:38:42.232568 23213 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:38:42.232370 23113 master_runner.cc:186] Error getting master registration for 127.0.98.62:37197: OK, instance_id { permanent_uuid: "389bf0585a2c4187bc2feb7f262f29a6" instance_seqno: 1790563121913840 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:38:42.234465 23113 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:38:42.254078 23031 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36541 } attempt: 2
Sep 28 02:38:42 dist-test-slave-fz47 krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 02:38:42.262851 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:42.254455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36541 (local address 127.0.0.1:33426)
0928 02:38:42.254517 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 02:38:42.254528 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 02:38:42.254570 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:38:42.254864 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:38:42.254865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:38:42.255055 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:38:42.255116 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:42.255118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.256559 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.256560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:42.257436 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:38:42.257439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.257912 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.257913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:38:42.257968 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:42.258448 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:38:42.258454 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:38:42.259940 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:38:42.261732 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:42.261733 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:42.261734 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:42.261960 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:42.262155 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:38:42.262168 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:38:42.262168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:38:42.262233 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:38:42.262527 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:38:42.262528 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:38:42.262577 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 02:38:42.262648 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 02:38:42.262745 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 02:38:42.263404 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:42.254660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33426 (local address 127.0.98.60:36541)
0928 02:38:42.254720 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:42.254722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:42.254731 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:38:42.254756 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:42.254757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:42.254767 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:42.254838 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:42.255209 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.256475 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.257520 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.257827 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.257900 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:42.260035 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:42.260037 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:42.260037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:42.260044 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:42.261646 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:42.262040 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:42.262040 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:42.262041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:42.262103 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:42.262295 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:42.262296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:42.262296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:42.262436 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:42.262496 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:42.263233 (+ 737us) server_negotiation.cc:299] Negotiation successful
0928 02:38:42.263313 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 02:38:42.265154 23031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 389bf0585a2c4187bc2feb7f262f29a6: Bootstrap starting.
I20260928 02:38:42.265985 23031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 389bf0585a2c4187bc2feb7f262f29a6: Neither blocks nor log segments found. Creating new log.
I20260928 02:38:42.266278 23031 log.cc:824] T 00000000000000000000000000000000 P 389bf0585a2c4187bc2feb7f262f29a6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:38:42.267143 23031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 389bf0585a2c4187bc2feb7f262f29a6: No bootstrap required, opened a new log
I20260928 02:38:42.268476 23031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 389bf0585a2c4187bc2feb7f262f29a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389bf0585a2c4187bc2feb7f262f29a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37197 } } peers { permanent_uuid: "9581fe9a54934139a6aee3eefa77618f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41365 } } peers { permanent_uuid: "c80e2d2bef0b41d7a2c1cb8f86869144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36541 } }
I20260928 02:38:42.268610 23031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 389bf0585a2c4187bc2feb7f262f29a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 389bf0585a2c4187bc2feb7f262f29a6, State: Initialized, Role: FOLLOWER
I20260928 02:38:42.268697 23031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 389bf0585a2c4187bc2feb7f262f29a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389bf0585a2c4187bc2feb7f262f29a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37197 } } peers { permanent_uuid: "9581fe9a54934139a6aee3eefa77618f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41365 } } peers { permanent_uuid: "c80e2d2bef0b41d7a2c1cb8f86869144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36541 } }
I20260928 02:38:42.268978 23216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 389bf0585a2c4187bc2feb7f262f29a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389bf0585a2c4187bc2feb7f262f29a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37197 } } peers { permanent_uuid: "9581fe9a54934139a6aee3eefa77618f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41365 } } peers { permanent_uuid: "c80e2d2bef0b41d7a2c1cb8f86869144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36541 } } }
I20260928 02:38:42.269011 23005 raft_consensus.cc:482] T 00000000000000000000000000000000 P 389bf0585a2c4187bc2feb7f262f29a6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 02:38:42.269208 23216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 389bf0585a2c4187bc2feb7f262f29a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:38:42.269246 23005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 389bf0585a2c4187bc2feb7f262f29a6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:42.270032 23005 raft_consensus.cc:504] T 00000000000000000000000000000000 P 389bf0585a2c4187bc2feb7f262f29a6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389bf0585a2c4187bc2feb7f262f29a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37197 } } peers { permanent_uuid: "9581fe9a54934139a6aee3eefa77618f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41365 } } peers { permanent_uuid: "c80e2d2bef0b41d7a2c1cb8f86869144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36541 } }
I20260928 02:38:42.270260 23005 leader_election.cc:288] T 00000000000000000000000000000000 P 389bf0585a2c4187bc2feb7f262f29a6 [CANDIDATE]: Term 1 election: Requested vote from peers 9581fe9a54934139a6aee3eefa77618f (127.0.98.61:41365), c80e2d2bef0b41d7a2c1cb8f86869144 (127.0.98.60:36541)
I20260928 02:38:42.268992 23031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 389bf0585a2c4187bc2feb7f262f29a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:38:42.270684 23080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "389bf0585a2c4187bc2feb7f262f29a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9581fe9a54934139a6aee3eefa77618f"
I20260928 02:38:42.270813 23080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9581fe9a54934139a6aee3eefa77618f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:38:42.272068 23080 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9581fe9a54934139a6aee3eefa77618f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 389bf0585a2c4187bc2feb7f262f29a6 in term 1.
I20260928 02:38:42.272368 23155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "389bf0585a2c4187bc2feb7f262f29a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c80e2d2bef0b41d7a2c1cb8f86869144"
I20260928 02:38:42.272498 23155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c80e2d2bef0b41d7a2c1cb8f86869144 [term 0 FOLLOWER]: Advancing to term 1
W20260928 02:38:42.272679 23231 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 389bf0585a2c4187bc2feb7f262f29a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:38:42.272724 23231 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:38:42.273437 22981 leader_election.cc:302] T 00000000000000000000000000000000 P 389bf0585a2c4187bc2feb7f262f29a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 389bf0585a2c4187bc2feb7f262f29a6, 9581fe9a54934139a6aee3eefa77618f; no voters:
I20260928 02:38:42.274204 23155 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c80e2d2bef0b41d7a2c1cb8f86869144 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 389bf0585a2c4187bc2feb7f262f29a6 in term 1.
I20260928 02:38:42.275640 22963 master_runner.cc:263] No leader master found from master 389bf0585a2c4187bc2feb7f262f29a6
I20260928 02:38:42.275681 22963 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:38:42.275769 23216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 389bf0585a2c4187bc2feb7f262f29a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:38:42.275933 23216 raft_consensus.cc:686] T 00000000000000000000000000000000 P 389bf0585a2c4187bc2feb7f262f29a6 [term 1 LEADER]: Becoming Leader. State: Replica: 389bf0585a2c4187bc2feb7f262f29a6, State: Running, Role: LEADER
I20260928 02:38:42.276051 23216 consensus_queue.cc:237] T 00000000000000000000000000000000 P 389bf0585a2c4187bc2feb7f262f29a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389bf0585a2c4187bc2feb7f262f29a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37197 } } peers { permanent_uuid: "9581fe9a54934139a6aee3eefa77618f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41365 } } peers { permanent_uuid: "c80e2d2bef0b41d7a2c1cb8f86869144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36541 } }
I20260928 02:38:42.276511 23216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 389bf0585a2c4187bc2feb7f262f29a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 389bf0585a2c4187bc2feb7f262f29a6. Latest consensus state: current_term: 1 leader_uuid: "389bf0585a2c4187bc2feb7f262f29a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389bf0585a2c4187bc2feb7f262f29a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37197 } } peers { permanent_uuid: "9581fe9a54934139a6aee3eefa77618f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41365 } } peers { permanent_uuid: "c80e2d2bef0b41d7a2c1cb8f86869144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36541 } } }
I20260928 02:38:42.276646 23216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 389bf0585a2c4187bc2feb7f262f29a6 [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:42.280092 23238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:38:42.280329 23238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:38:42.286356 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:42.273009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39856 (local address 127.0.98.62:37197)
0928 02:38:42.273045 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:42.273047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:42.273955 (+ 908us) server_negotiation.cc:407] Connection header received
0928 02:38:42.274879 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:42.274880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:42.274890 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:42.274918 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:42.276441 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.277143 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.279109 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.279398 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.279435 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:42.282416 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:42.282417 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:42.282417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:42.282424 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:42.284460 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:42.285158 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:42.285159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:42.285159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:42.285205 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:42.285475 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:42.285476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:42.285477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:42.285588 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:42.285670 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:42.286199 (+ 529us) server_negotiation.cc:299] Negotiation successful
0928 02:38:42.286273 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:38:42.287374 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:42.273185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33434 (local address 127.0.98.60:36541)
0928 02:38:42.273219 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:42.273220 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:42.273869 (+ 649us) server_negotiation.cc:407] Connection header received
0928 02:38:42.274048 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:42.274049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:42.274059 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:42.274086 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:42.275603 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.276343 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.279776 (+ 3433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.280066 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.280103 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:42.282144 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:42.282146 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:42.282146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:42.282153 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:42.283810 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:42.285035 (+ 1225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:42.285036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:42.285037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:42.285085 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:42.285822 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:42.285823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:42.285824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:42.285935 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:42.286000 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:42.287215 (+ 1215us) server_negotiation.cc:299] Negotiation successful
0928 02:38:42.287287 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 02:38:42.289104 23155 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c80e2d2bef0b41d7a2c1cb8f86869144 [term 1 FOLLOWER]: Refusing update from remote peer 389bf0585a2c4187bc2feb7f262f29a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:38:42.289497 23216 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 389bf0585a2c4187bc2feb7f262f29a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c80e2d2bef0b41d7a2c1cb8f86869144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:38:42.290776 23202 sys_catalog.cc:455] T 00000000000000000000000000000000 P c80e2d2bef0b41d7a2c1cb8f86869144 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 389bf0585a2c4187bc2feb7f262f29a6. Latest consensus state: current_term: 1 leader_uuid: "389bf0585a2c4187bc2feb7f262f29a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389bf0585a2c4187bc2feb7f262f29a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37197 } } peers { permanent_uuid: "9581fe9a54934139a6aee3eefa77618f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41365 } } peers { permanent_uuid: "c80e2d2bef0b41d7a2c1cb8f86869144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36541 } } }
I20260928 02:38:42.290858 23202 sys_catalog.cc:458] T 00000000000000000000000000000000 P c80e2d2bef0b41d7a2c1cb8f86869144 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:38:42.291064 23080 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9581fe9a54934139a6aee3eefa77618f [term 1 FOLLOWER]: Refusing update from remote peer 389bf0585a2c4187bc2feb7f262f29a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:38:42.291467 23216 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 389bf0585a2c4187bc2feb7f262f29a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9581fe9a54934139a6aee3eefa77618f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:38:42.292550 23188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9581fe9a54934139a6aee3eefa77618f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 389bf0585a2c4187bc2feb7f262f29a6. Latest consensus state: current_term: 1 leader_uuid: "389bf0585a2c4187bc2feb7f262f29a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389bf0585a2c4187bc2feb7f262f29a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37197 } } peers { permanent_uuid: "9581fe9a54934139a6aee3eefa77618f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41365 } } peers { permanent_uuid: "c80e2d2bef0b41d7a2c1cb8f86869144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36541 } } }
I20260928 02:38:42.292629 23188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9581fe9a54934139a6aee3eefa77618f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:38:42.294860 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:42.273585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60942 (local address 127.0.98.61:41365)
0928 02:38:42.273621 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:42.273623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:42.288041 (+ 14418us) server_negotiation.cc:407] Connection header received
0928 02:38:42.288127 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:42.288128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:42.288139 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:42.288182 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:42.288822 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.289533 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.290511 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:42.290787 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:42.290827 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:42.292144 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:38:42.292146 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:38:42.292146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:38:42.292153 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:38:42.293658 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:42.294024 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:42.294026 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:42.294026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:42.294080 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:38:42.294284 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:38:42.294285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:38:42.294285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:38:42.294410 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:38:42.294479 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:42.294704 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 02:38:42.294777 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 02:38:42.295991 23216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 389bf0585a2c4187bc2feb7f262f29a6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "389bf0585a2c4187bc2feb7f262f29a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389bf0585a2c4187bc2feb7f262f29a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37197 } } peers { permanent_uuid: "9581fe9a54934139a6aee3eefa77618f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41365 } } peers { permanent_uuid: "c80e2d2bef0b41d7a2c1cb8f86869144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36541 } } }
I20260928 02:38:42.296072 23216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 389bf0585a2c4187bc2feb7f262f29a6 [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:42.296057 23188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9581fe9a54934139a6aee3eefa77618f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "389bf0585a2c4187bc2feb7f262f29a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389bf0585a2c4187bc2feb7f262f29a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37197 } } peers { permanent_uuid: "9581fe9a54934139a6aee3eefa77618f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41365 } } peers { permanent_uuid: "c80e2d2bef0b41d7a2c1cb8f86869144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36541 } } }
I20260928 02:38:42.296141 23188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9581fe9a54934139a6aee3eefa77618f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:38:42.297012 23239 mvcc.cc:205] Tried to move back new op lower bound from 7334146548894023680 to 7334146548843827200. Current Snapshot: MvccSnapshot[applied={T|T < 7334146548894023680}]
I20260928 02:38:42.297514 23242 mvcc.cc:205] Tried to move back new op lower bound from 7334146548894023680 to 7334146548843827200. Current Snapshot: MvccSnapshot[applied={T|T < 7334146548894023680}]
I20260928 02:38:42.298177 23236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 389bf0585a2c4187bc2feb7f262f29a6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "389bf0585a2c4187bc2feb7f262f29a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389bf0585a2c4187bc2feb7f262f29a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37197 } } peers { permanent_uuid: "9581fe9a54934139a6aee3eefa77618f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41365 } } peers { permanent_uuid: "c80e2d2bef0b41d7a2c1cb8f86869144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36541 } } }
I20260928 02:38:42.298264 23236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 389bf0585a2c4187bc2feb7f262f29a6 [sys.catalog]: This master's current role is: LEADER
I20260928 02:38:42.298658 23202 sys_catalog.cc:455] T 00000000000000000000000000000000 P c80e2d2bef0b41d7a2c1cb8f86869144 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "389bf0585a2c4187bc2feb7f262f29a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389bf0585a2c4187bc2feb7f262f29a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37197 } } peers { permanent_uuid: "9581fe9a54934139a6aee3eefa77618f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41365 } } peers { permanent_uuid: "c80e2d2bef0b41d7a2c1cb8f86869144" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36541 } } }
I20260928 02:38:42.298738 23202 sys_catalog.cc:458] T 00000000000000000000000000000000 P c80e2d2bef0b41d7a2c1cb8f86869144 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:38:42.298894 23240 mvcc.cc:205] Tried to move back new op lower bound from 7334146548894023680 to 7334146548843827200. Current Snapshot: MvccSnapshot[applied={T|T < 7334146548894023680}]
I20260928 02:38:42.304883 23238 catalog_manager.cc:1347] Generated new cluster ID: e7905dbb6f774bf594042853a926dd46
I20260928 02:38:42.304941 23238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:38:42.372721 23238 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:38:42.373462 23238 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:38:42.381377 23238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 389bf0585a2c4187bc2feb7f262f29a6: Generated new TSK 0
I20260928 02:38:42.381548 23238 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:38:42.405872 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:38:42.404970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33438 (local address 127.0.98.60:36541)
0928 02:38:42.405016 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:42.405018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:42.405176 (+ 158us) server_negotiation.cc:407] Connection header received
0928 02:38:42.405310 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:42.405311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:42.405322 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:42.405351 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:42.405804 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33438: BlockingRecv error: recv got EOF from 127.0.0.1:33438 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 02:38:42.405871 23034 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:38:42.405064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39870 (local address 127.0.98.62:37197)
0928 02:38:42.405104 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:42.405106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:42.405202 (+ 96us) server_negotiation.cc:407] Connection header received
0928 02:38:42.405310 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:42.405311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:42.405322 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:42.405351 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:42.405804 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39870: BlockingRecv error: recv got EOF from 127.0.0.1:39870 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 02:38:42.406075 23110 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:38:42.405230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60958 (local address 127.0.98.61:41365)
0928 02:38:42.405294 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:42.405296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:42.405508 (+ 212us) server_negotiation.cc:407] Connection header received
0928 02:38:42.405609 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:42.405610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:42.405620 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:42.405649 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:42.406016 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60958: BlockingRecv error: recv got EOF from 127.0.0.1:60958 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 02:38:42.408022 23110 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:38:42.407510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60972 (local address 127.0.98.61:41365)
0928 02:38:42.407577 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:42.407579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:42.407600 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:38:42.407669 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:42.407669 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:42.407685 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:42.407729 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:42.407981 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60972: BlockingRecv error: recv got EOF from 127.0.0.1:60972 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 02:38:42.410566 23110 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:38:42.410126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60982 (local address 127.0.98.61:41365)
0928 02:38:42.410185 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:42.410186 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:42.410194 (+ 8us) server_negotiation.cc:407] Connection header received
0928 02:38:42.410213 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:42.410214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:42.410222 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:42.410279 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:42.410524 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60982: BlockingRecv error: recv got EOF from 127.0.0.1:60982 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 02:38:42.414109 23110 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:38:42.413629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60996 (local address 127.0.98.61:41365)
0928 02:38:42.413701 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:42.413703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:42.413712 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:38:42.413732 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:42.413733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:42.413742 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:42.413801 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:42.414066 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60996: BlockingRecv error: recv got EOF from 127.0.0.1:60996 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 02:38:42.419477 23110 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:38:42.418921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32778 (local address 127.0.98.61:41365)
0928 02:38:42.418967 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:42.418969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:42.419075 (+ 106us) server_negotiation.cc:407] Connection header received
0928 02:38:42.419133 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:42.419133 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:42.419140 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:42.419182 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:42.419436 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32778: BlockingRecv error: recv got EOF from 127.0.0.1:32778 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260928 02:38:42.429095 23110 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:38:42.428484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32784 (local address 127.0.98.61:41365)
0928 02:38:42.428521 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:42.428523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:42.428653 (+ 130us) server_negotiation.cc:407] Connection header received
0928 02:38:42.428692 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:42.428693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:42.428701 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:42.428756 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:42.429047 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32784: BlockingRecv error: recv got EOF from 127.0.0.1:32784 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 02:38:42.446776 23110 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:38:42.446180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32798 (local address 127.0.98.61:41365)
0928 02:38:42.446225 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:42.446227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:42.446315 (+ 88us) server_negotiation.cc:407] Connection header received
0928 02:38:42.446348 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:42.446349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:42.446359 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:42.446422 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:42.446720 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32798: BlockingRecv error: recv got EOF from 127.0.0.1:32798 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 02:38:42.480404 23110 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:38:42.479803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32808 (local address 127.0.98.61:41365)
0928 02:38:42.479897 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:42.479899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:42.479947 (+ 48us) server_negotiation.cc:407] Connection header received
0928 02:38:42.479992 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:42.479993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:42.480003 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:42.480051 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:42.480342 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32808: BlockingRecv error: recv got EOF from 127.0.0.1:32808 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 02:38:42.546252 23110 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:38:42.545541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32824 (local address 127.0.98.61:41365)
0928 02:38:42.545632 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:42.545635 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:42.545718 (+ 83us) server_negotiation.cc:407] Connection header received
0928 02:38:42.545781 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:42.545781 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:42.545793 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:42.545840 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:42.546185 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32824: BlockingRecv error: recv got EOF from 127.0.0.1:32824 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 02:38:42.676033 23110 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:38:42.675293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32828 (local address 127.0.98.61:41365)
0928 02:38:42.675353 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:42.675355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:42.675455 (+ 100us) server_negotiation.cc:407] Connection header received
0928 02:38:42.675575 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:42.675576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:42.675589 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:42.675636 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:42.675960 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32828: BlockingRecv error: recv got EOF from 127.0.0.1:32828 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260928 02:38:42.933658 23110 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:38:42.933068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32838 (local address 127.0.98.61:41365)
0928 02:38:42.933130 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:42.933132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:42.933141 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:38:42.933188 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:42.933189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:42.933199 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:42.933243 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:38:42.933597 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32838: BlockingRecv error: recv got EOF from 127.0.0.1:32838 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:38:43.216260 23199 catalog_manager.cc:1259] Loaded cluster ID: e7905dbb6f774bf594042853a926dd46
I20260928 02:38:43.216331 23199 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9581fe9a54934139a6aee3eefa77618f: loading cluster ID for follower catalog manager: success
I20260928 02:38:43.217131 23199 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9581fe9a54934139a6aee3eefa77618f: acquiring CA information for follower catalog manager: success
I20260928 02:38:43.217447 23199 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9581fe9a54934139a6aee3eefa77618f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:38:43.232987 23213 catalog_manager.cc:1259] Loaded cluster ID: e7905dbb6f774bf594042853a926dd46
I20260928 02:38:43.233050 23213 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c80e2d2bef0b41d7a2c1cb8f86869144: loading cluster ID for follower catalog manager: success
I20260928 02:38:43.233866 23213 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c80e2d2bef0b41d7a2c1cb8f86869144: acquiring CA information for follower catalog manager: success
I20260928 02:38:43.234158 23213 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c80e2d2bef0b41d7a2c1cb8f86869144: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:38:43.449029 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:43.446728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32842 (local address 127.0.98.61:41365)
0928 02:38:43.446819 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:43.446821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:43.446832 (+ 11us) server_negotiation.cc:407] Connection header received
0928 02:38:43.446887 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:43.446888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:43.446936 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:43.447015 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:38:43.447136 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:43.447620 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:43.448346 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:43.448577 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:43.448628 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:43.448764 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:38:43.448868 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:38:43.448919 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:43.448947 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 02:38:43.448962 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":46,"threads_started":1}
I20260928 02:38:43.453047 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:43.450712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33442 (local address 127.0.98.60:36541)
0928 02:38:43.450814 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:43.450815 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:43.450824 (+ 9us) server_negotiation.cc:407] Connection header received
0928 02:38:43.450870 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:43.450870 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:43.450904 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:43.450958 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:38:43.451035 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:43.451489 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:43.452131 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:43.452411 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:43.452471 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:43.452671 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:38:43.452790 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:38:43.452861 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:43.452925 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 02:38:43.452962 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":58,"threads_started":1}
I20260928 02:38:43.457345 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:43.454800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33446 (local address 127.0.98.60:36541)
0928 02:38:43.454854 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:43.454856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:43.455163 (+ 307us) server_negotiation.cc:407] Connection header received
0928 02:38:43.455226 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:43.455227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:43.455237 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:43.455282 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:38:43.455470 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:43.455988 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:43.456794 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:43.456961 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:43.457007 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:43.457135 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:38:43.457156 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:38:43.457190 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:43.457240 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 02:38:43.457275 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 02:38:43.457916 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:43.454971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32846 (local address 127.0.98.61:41365)
0928 02:38:43.455004 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:43.455006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:43.455401 (+ 395us) server_negotiation.cc:407] Connection header received
0928 02:38:43.455457 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:43.455458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:43.455469 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:43.455519 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:38:43.455650 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:43.456345 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:43.457201 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:43.457480 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:43.457535 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:43.457719 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:38:43.457751 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:38:43.457799 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:43.457832 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 02:38:43.457851 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 02:38:43.458703 23324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:38:43.454805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39874 (local address 127.0.98.62:37197)
0928 02:38:43.454907 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 02:38:43.454908 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 02:38:43.455057 (+ 149us) server_negotiation.cc:407] Connection header received
0928 02:38:43.455159 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:38:43.455160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:38:43.455210 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:38:43.455284 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:38:43.455596 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:43.456351 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:43.458040 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:38:43.458302 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:38:43.458346 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:38:43.458477 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:38:43.458543 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:38:43.458585 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 02:38:43.458612 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 02:38:43.458628 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":51,"threads_started":1}
I20260928 02:38:43.460147 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 22963
I20260928 02:38:43.464563 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 23038
I20260928 02:38:43.468636 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskyt1071/build/release/bin/kudu with pid 23113
2026-09-28T02:38:43Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2746 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5419 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (260669 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST